[==========] Running 56 tests from 4 test suites.
[----------] Global test environment set-up.
[----------] 30 tests from SecurityITest
[ RUN ] SecurityITest.TestAuthorizationOnListTablets
Loading random data
Initializing database '/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790722213368738-14927-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 22:50:13 dist-test-slave-98mk krb5kdc[14934](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 22:50:13 dist-test-slave-98mk krb5kdc[14934](info): set up 2 sockets
Sep 29 22:50:13 dist-test-slave-98mk krb5kdc[14934](info): commencing operation
krb5kdc: starting...
WARNING: Logging before InitGoogleLogging() is written to STDERR
W20260929 22:50:14.331835 14927 mini_kdc.cc:121] Time spent starting KDC: real 0.945s user 0.002s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790722213368738-14927-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790722213368738-14927-0/krb5kdc/test-user.keytab.
Sep 29 22:50:14 dist-test-slave-98mk krb5kdc[14934](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790722214, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T22:50:14Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T22:50:14Z Disabled control of system clock
WARNING: no policy specified for kudu/127.14.147.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.147.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.147.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790722213368738-14927-0/krb5kdc/kudu_127.14.147.254.keytab.
Entry for principal kudu/127.14.147.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790722213368738-14927-0/krb5kdc/kudu_127.14.147.254.keytab.
I20260929 22:50:14.452466 14927 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhWOCLk/build/release/bin/kudu
/tmp/dist-test-taskhWOCLk/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790722213368738-14927-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790722213368738-14927-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790722213368738-14927-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790722213368738-14927-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.147.254:40593
--webserver_interface=127.14.147.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790722213368738-14927-0/krb5kdc/kudu_127.14.147.254.keytab
--principal=kudu/127.14.147.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.147.212:34039
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.147.254:40593
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790722213368738-14927-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790722213368738-14927-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790722213368738-14927-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 22:50:14.532119 14947 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 22:50:14.532348 14947 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 22:50:14.532379 14947 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 22:50:14.533603 14947 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 22:50:14.533648 14947 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 22:50:14.533664 14947 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 22:50:14.533679 14947 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 22:50:14.533694 14947 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 22:50:14.535205 14947 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.147.212:34039
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790722213368738-14927-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790722213368738-14927-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.14.147.254:40593
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790722213368738-14927-0/krb5kdc/kudu_127.14.147.254.keytab
--principal=kudu/127.14.147.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.147.254:40593
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790722213368738-14927-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.14.147.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14947
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790722213368738-14927-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 21:58:54 UTC on ccycloud.quasar-xguoti.root.comops.site
build id 4681390
I20260929 22:50:14.535507 14947 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 22:50:14.535724 14947 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 22:50:14.540745 14953 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 22:50:14.540755 14952 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 22:50:14.540792 14947 server_base.cc:1034] running on GCE node
W20260929 22:50:14.540755 14955 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 22:50:14.541175 14947 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 22:50:14.541501 14947 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 22:50:14.542659 14947 hybrid_clock.cc:648] HybridClock initialized: now 1790722214542646 us; error 28 us; skew 500 ppm
Sep 29 22:50:14 dist-test-slave-98mk krb5kdc[14934](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790722214, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.147.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 22:50:14.545668 14947 init.cc:377] Logged in from keytab as kudu/127.14.147.254@KRBTEST.COM (short username kudu)
I20260929 22:50:14.546120 14947 webserver.cc:460] Webserver started at http://127.14.147.254:36875/ using document root <none> and password file <none>
I20260929 22:50:14.546324 14947 fs_manager.cc:359] Metadata directory not provided
I20260929 22:50:14.546375 14947 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 22:50:14.546500 14947 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 22:50:14.547672 14947 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790722213368738-14927-0/minicluster-data/master-0/data/instance:
uuid: "37f8fc1afd42473f82b1f6d808f1474d"
format_stamp: "Formatted at 2026-09-29 22:50:14 on dist-test-slave-98mk"
server_key: "07e0560394d0fceec227bbf88134bcc2"
server_key_iv: "e9dcbaeacb5dc2babe69c3b25753a0ac"
server_key_version: "encryptionkey@0"
I20260929 22:50:14.548017 14947 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790722213368738-14927-0/minicluster-data/master-0/wal/instance:
uuid: "37f8fc1afd42473f82b1f6d808f1474d"
format_stamp: "Formatted at 2026-09-29 22:50:14 on dist-test-slave-98mk"
server_key: "07e0560394d0fceec227bbf88134bcc2"
server_key_iv: "e9dcbaeacb5dc2babe69c3b25753a0ac"
server_key_version: "encryptionkey@0"
I20260929 22:50:14.549484 14947 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260929 22:50:14.550346 14962 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 22:50:14.550594 14947 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260929 22:50:14.550693 14947 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790722213368738-14927-0/minicluster-data/master-0/data,/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790722213368738-14927-0/minicluster-data/master-0/wal
uuid: "37f8fc1afd42473f82b1f6d808f1474d"
format_stamp: "Formatted at 2026-09-29 22:50:14 on dist-test-slave-98mk"
server_key: "07e0560394d0fceec227bbf88134bcc2"
server_key_iv: "e9dcbaeacb5dc2babe69c3b25753a0ac"
server_key_version: "encryptionkey@0"
I20260929 22:50:14.550760 14947 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790722213368738-14927-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790722213368738-14927-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790722213368738-14927-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 22:50:14.579550 14947 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 22:50:14.581770 14947 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 22:50:14.581919 14947 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 22:50:14.585790 14947 rpc_server.cc:307] RPC server started. Bound to: 127.14.147.254:40593
I20260929 22:50:14.585944 15015 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.147.254:40593 every 8 connection(s)
I20260929 22:50:14.586184 14947 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790722213368738-14927-0/minicluster-data/master-0/data/info.pb
I20260929 22:50:14.587037 15016 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 22:50:14.587735 14927 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhWOCLk/build/release/bin/kudu as pid 14947
I20260929 22:50:14.587824 14927 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790722213368738-14927-0/minicluster-data/master-0/wal/instance
I20260929 22:50:14.588488 14927 external_mini_cluster.cc:1428] Setting key 2dca7c29befad6c4e80d91d2ab1e96e8
I20260929 22:50:14.589706 15016 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 37f8fc1afd42473f82b1f6d808f1474d: Bootstrap starting.
I20260929 22:50:14.590272 15016 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 37f8fc1afd42473f82b1f6d808f1474d: Neither blocks nor log segments found. Creating new log.
I20260929 22:50:14.590526 15016 log.cc:824] T 00000000000000000000000000000000 P 37f8fc1afd42473f82b1f6d808f1474d: Log is configured to *not* fsync() on all Append() calls
I20260929 22:50:14.591171 15016 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 37f8fc1afd42473f82b1f6d808f1474d: No bootstrap required, opened a new log
I20260929 22:50:14.593068 15016 raft_consensus.cc:348] T 00000000000000000000000000000000 P 37f8fc1afd42473f82b1f6d808f1474d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "37f8fc1afd42473f82b1f6d808f1474d" member_type: VOTER last_known_addr { host: "127.14.147.254" port: 40593 } }
I20260929 22:50:14.593209 15016 raft_consensus.cc:374] T 00000000000000000000000000000000 P 37f8fc1afd42473f82b1f6d808f1474d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 22:50:14.593243 15016 raft_consensus.cc:729] T 00000000000000000000000000000000 P 37f8fc1afd42473f82b1f6d808f1474d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 37f8fc1afd42473f82b1f6d808f1474d, State: Initialized, Role: FOLLOWER
I20260929 22:50:14.593367 15016 consensus_queue.cc:260] T 00000000000000000000000000000000 P 37f8fc1afd42473f82b1f6d808f1474d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "37f8fc1afd42473f82b1f6d808f1474d" member_type: VOTER last_known_addr { host: "127.14.147.254" port: 40593 } }
I20260929 22:50:14.593439 15016 raft_consensus.cc:388] T 00000000000000000000000000000000 P 37f8fc1afd42473f82b1f6d808f1474d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 22:50:14.593472 15016 raft_consensus.cc:482] T 00000000000000000000000000000000 P 37f8fc1afd42473f82b1f6d808f1474d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 22:50:14.593519 15016 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 37f8fc1afd42473f82b1f6d808f1474d [term 0 FOLLOWER]: Advancing to term 1
I20260929 22:50:14.594398 15016 raft_consensus.cc:504] T 00000000000000000000000000000000 P 37f8fc1afd42473f82b1f6d808f1474d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "37f8fc1afd42473f82b1f6d808f1474d" member_type: VOTER last_known_addr { host: "127.14.147.254" port: 40593 } }
I20260929 22:50:14.594501 15016 leader_election.cc:302] T 00000000000000000000000000000000 P 37f8fc1afd42473f82b1f6d808f1474d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 37f8fc1afd42473f82b1f6d808f1474d; no voters:
I20260929 22:50:14.594673 15016 leader_election.cc:288] T 00000000000000000000000000000000 P 37f8fc1afd42473f82b1f6d808f1474d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 22:50:14.594769 15021 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 37f8fc1afd42473f82b1f6d808f1474d [term 1 FOLLOWER]: Leader election won for term 1
I20260929 22:50:14.594897 15021 raft_consensus.cc:686] T 00000000000000000000000000000000 P 37f8fc1afd42473f82b1f6d808f1474d [term 1 LEADER]: Becoming Leader. State: Replica: 37f8fc1afd42473f82b1f6d808f1474d, State: Running, Role: LEADER
I20260929 22:50:14.595019 15016 sys_catalog.cc:564] T 00000000000000000000000000000000 P 37f8fc1afd42473f82b1f6d808f1474d [sys.catalog]: configured and running, proceeding with master startup.
I20260929 22:50:14.594986 15021 consensus_queue.cc:237] T 00000000000000000000000000000000 P 37f8fc1afd42473f82b1f6d808f1474d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "37f8fc1afd42473f82b1f6d808f1474d" member_type: VOTER last_known_addr { host: "127.14.147.254" port: 40593 } }
I20260929 22:50:14.595371 15021 sys_catalog.cc:455] T 00000000000000000000000000000000 P 37f8fc1afd42473f82b1f6d808f1474d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 37f8fc1afd42473f82b1f6d808f1474d. Latest consensus state: current_term: 1 leader_uuid: "37f8fc1afd42473f82b1f6d808f1474d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "37f8fc1afd42473f82b1f6d808f1474d" member_type: VOTER last_known_addr { host: "127.14.147.254" port: 40593 } } }
I20260929 22:50:14.595469 15021 sys_catalog.cc:458] T 00000000000000000000000000000000 P 37f8fc1afd42473f82b1f6d808f1474d [sys.catalog]: This master's current role is: LEADER
I20260929 22:50:14.595677 15023 sys_catalog.cc:455] T 00000000000000000000000000000000 P 37f8fc1afd42473f82b1f6d808f1474d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "37f8fc1afd42473f82b1f6d808f1474d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "37f8fc1afd42473f82b1f6d808f1474d" member_type: VOTER last_known_addr { host: "127.14.147.254" port: 40593 } } }
I20260929 22:50:14.595759 15023 sys_catalog.cc:458] T 00000000000000000000000000000000 P 37f8fc1afd42473f82b1f6d808f1474d [sys.catalog]: This master's current role is: LEADER
I20260929 22:50:14.596231 15030 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 22:50:14.596751 15030 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 22:50:14.598225 15030 catalog_manager.cc:1347] Generated new cluster ID: 7bd2b80fd78e466d9466ff887e222741
I20260929 22:50:14.598282 15030 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 29 22:50:14 dist-test-slave-98mk krb5kdc[14934](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790722214, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.147.254@KRBTEST.COM
I20260929 22:50:14.606343 15018 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:50:14.589386 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38760 (local address 127.14.147.254:40593)
0929 22:50:14.589565 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0929 22:50:14.589575 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 22:50:14.589797 (+ 222us) server_negotiation.cc:407] Connection header received
0929 22:50:14.592355 (+ 2558us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 22:50:14.592362 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 22:50:14.592419 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 22:50:14.592548 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0929 22:50:14.594569 (+ 2021us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:50:14.596279 (+ 1710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:50:14.598008 (+ 1729us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:50:14.598535 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:50:14.598654 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 22:50:14.602096 (+ 3442us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 22:50:14.602101 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 22:50:14.602101 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 22:50:14.602124 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0929 22:50:14.604355 (+ 2231us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 22:50:14.604792 (+ 437us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 22:50:14.604796 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 22:50:14.604796 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 22:50:14.604890 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 22:50:14.605149 (+ 259us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 22:50:14.605150 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 22:50:14.605151 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 22:50:14.605397 (+ 246us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 22:50:14.605554 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0929 22:50:14.605843 (+ 289us) server_negotiation.cc:299] Negotiation successful
0929 22:50:14.605977 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":125,"thread_start_us":79,"threads_started":1}
I20260929 22:50:14.618937 15030 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 22:50:14.619546 15030 catalog_manager.cc:1504] Loading token signing keys...
I20260929 22:50:14.635367 15030 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 37f8fc1afd42473f82b1f6d808f1474d: Generated new TSK 0
I20260929 22:50:14.635609 15030 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.14.147.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.147.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.147.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790722213368738-14927-0/krb5kdc/kudu_127.14.147.193.keytab.
Entry for principal kudu/127.14.147.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790722213368738-14927-0/krb5kdc/kudu_127.14.147.193.keytab.
I20260929 22:50:14.670362 14927 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhWOCLk/build/release/bin/kudu
/tmp/dist-test-taskhWOCLk/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790722213368738-14927-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790722213368738-14927-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790722213368738-14927-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790722213368738-14927-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.147.193:0
--local_ip_for_outbound_sockets=127.14.147.193
--webserver_interface=127.14.147.193
--webserver_port=0
--tserver_master_addrs=127.14.147.254:40593
--keytab_file=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790722213368738-14927-0/krb5kdc/kudu_127.14.147.193.keytab
--principal=kudu/127.14.147.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.147.212:34039
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790722213368738-14927-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790722213368738-14927-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790722213368738-14927-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 22:50:14.770861 15042 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 22:50:14.771087 15042 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 22:50:14.771122 15042 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 22:50:14.772347 15042 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 22:50:14.772397 15042 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 22:50:14.772435 15042 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.147.193
I20260929 22:50:14.773959 15042 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.147.212:34039
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790722213368738-14927-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790722213368738-14927-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790722213368738-14927-0/krb5kdc/kudu_127.14.147.193.keytab
--principal=kudu/127.14.147.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.147.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790722213368738-14927-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.14.147.193
--webserver_port=0
--tserver_master_addrs=127.14.147.254:40593
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15042
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.147.193
--log_dir=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790722213368738-14927-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 21:58:54 UTC on ccycloud.quasar-xguoti.root.comops.site
build id 4681390
I20260929 22:50:14.774210 15042 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 22:50:14.774428 15042 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 22:50:14.779520 15048 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 22:50:14.779515 15047 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 22:50:14.779515 15050 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 22:50:14.779853 15042 server_base.cc:1034] running on GCE node
I20260929 22:50:14.780071 15042 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 22:50:14.780339 15042 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 22:50:14.781483 15042 hybrid_clock.cc:648] HybridClock initialized: now 1790722214781468 us; error 30 us; skew 500 ppm
Sep 29 22:50:14 dist-test-slave-98mk krb5kdc[14934](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790722214, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.147.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 22:50:14.784062 15042 init.cc:377] Logged in from keytab as kudu/127.14.147.193@KRBTEST.COM (short username kudu)
I20260929 22:50:14.784559 15042 webserver.cc:460] Webserver started at http://127.14.147.193:33995/ using document root <none> and password file <none>
I20260929 22:50:14.784777 15042 fs_manager.cc:359] Metadata directory not provided
I20260929 22:50:14.784827 15042 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 22:50:14.784956 15042 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 22:50:14.786088 15042 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790722213368738-14927-0/minicluster-data/ts-0/data/instance:
uuid: "c32daee579a64f5c9b70a6bbdc1ee569"
format_stamp: "Formatted at 2026-09-29 22:50:14 on dist-test-slave-98mk"
server_key: "be9c577555c11b67c1c34e3afea1369c"
server_key_iv: "376c85964aaf3cdbeeec9e52fa75636e"
server_key_version: "encryptionkey@0"
I20260929 22:50:14.786453 15042 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790722213368738-14927-0/minicluster-data/ts-0/wal/instance:
uuid: "c32daee579a64f5c9b70a6bbdc1ee569"
format_stamp: "Formatted at 2026-09-29 22:50:14 on dist-test-slave-98mk"
server_key: "be9c577555c11b67c1c34e3afea1369c"
server_key_iv: "376c85964aaf3cdbeeec9e52fa75636e"
server_key_version: "encryptionkey@0"
I20260929 22:50:14.787827 15042 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260929 22:50:14.788564 15057 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 22:50:14.788723 15042 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260929 22:50:14.788810 15042 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790722213368738-14927-0/minicluster-data/ts-0/data,/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790722213368738-14927-0/minicluster-data/ts-0/wal
uuid: "c32daee579a64f5c9b70a6bbdc1ee569"
format_stamp: "Formatted at 2026-09-29 22:50:14 on dist-test-slave-98mk"
server_key: "be9c577555c11b67c1c34e3afea1369c"
server_key_iv: "376c85964aaf3cdbeeec9e52fa75636e"
server_key_version: "encryptionkey@0"
I20260929 22:50:14.788872 15042 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790722213368738-14927-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790722213368738-14927-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790722213368738-14927-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 22:50:14.817180 15042 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 22:50:14.819496 15042 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 22:50:14.819661 15042 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 22:50:14.819903 15042 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 22:50:14.820219 15042 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 22:50:14.820255 15042 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 22:50:14.820284 15042 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 22:50:14.820302 15042 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 22:50:14.826586 15042 rpc_server.cc:307] RPC server started. Bound to: 127.14.147.193:36775
I20260929 22:50:14.826632 15170 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.147.193:36775 every 8 connection(s)
I20260929 22:50:14.826958 15042 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790722213368738-14927-0/minicluster-data/ts-0/data/info.pb
Sep 29 22:50:14 dist-test-slave-98mk krb5kdc[14934](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790722214, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.147.193@KRBTEST.COM for kudu/127.14.147.254@KRBTEST.COM
I20260929 22:50:14.836269 14927 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhWOCLk/build/release/bin/kudu as pid 15042
I20260929 22:50:14.836349 14927 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790722213368738-14927-0/minicluster-data/ts-0/wal/instance
I20260929 22:50:14.836478 14927 external_mini_cluster.cc:1428] Setting key 94b67d5f7feb314debe96410d48b1cb6
I20260929 22:50:14.836817 15018 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:50:14.827908 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.147.193:59509 (local address 127.14.147.254:40593)
0929 22:50:14.827996 (+ 88us) server_negotiation.cc:206] Beginning negotiation
0929 22:50:14.827998 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 22:50:14.828251 (+ 253us) server_negotiation.cc:407] Connection header received
0929 22:50:14.828789 (+ 538us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 22:50:14.828791 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 22:50:14.828804 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 22:50:14.828854 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0929 22:50:14.829868 (+ 1014us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:50:14.830392 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:50:14.831395 (+ 1003us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:50:14.831645 (+ 250us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:50:14.831706 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 22:50:14.834151 (+ 2445us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 22:50:14.834152 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 22:50:14.834153 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 22:50:14.834160 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0929 22:50:14.835316 (+ 1156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 22:50:14.835739 (+ 423us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 22:50:14.835740 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 22:50:14.835740 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 22:50:14.835796 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 22:50:14.836015 (+ 219us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 22:50:14.836016 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 22:50:14.836016 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 22:50:14.836142 (+ 126us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 22:50:14.836199 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0929 22:50:14.836624 (+ 425us) server_negotiation.cc:299] Negotiation successful
0929 22:50:14.836723 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260929 22:50:14.836997 15173 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:50:14.827868 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.147.254:40593 (local address 127.14.147.193:59509)
0929 22:50:14.828103 (+ 235us) negotiation.cc:107] Waiting for socket to connect
0929 22:50:14.828166 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0929 22:50:14.828687 (+ 521us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 22:50:14.828941 (+ 254us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 22:50:14.828946 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 22:50:14.829552 (+ 606us) client_negotiation.cc:189] Negotiated authn=SASL
0929 22:50:14.829764 (+ 212us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 22:50:14.829770 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 22:50:14.830482 (+ 712us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 22:50:14.830483 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 22:50:14.831304 (+ 821us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 22:50:14.831307 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 22:50:14.831724 (+ 417us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 22:50:14.831726 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 22:50:14.831862 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 22:50:14.832507 (+ 645us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 22:50:14.832515 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0929 22:50:14.834047 (+ 1532us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 22:50:14.835395 (+ 1348us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 22:50:14.835402 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 22:50:14.835405 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 22:50:14.835672 (+ 267us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 22:50:14.835851 (+ 179us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 22:50:14.835852 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 22:50:14.835852 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 22:50:14.835958 (+ 106us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 22:50:14.836198 (+ 240us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 22:50:14.836205 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 22:50:14.836468 (+ 263us) client_negotiation.cc:769] Sending connection context
0929 22:50:14.836586 (+ 118us) client_negotiation.cc:240] Negotiation successful
0929 22:50:14.836767 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":183,"thread_start_us":107,"threads_started":1}
I20260929 22:50:14.837489 15171 heartbeater.cc:347] Connected to a master server at 127.14.147.254:40593
I20260929 22:50:14.837596 15171 heartbeater.cc:464] Registering TS with master...
I20260929 22:50:14.837806 15171 heartbeater.cc:511] Master 127.14.147.254:40593 requested a full tablet report, sending...
I20260929 22:50:14.838335 14980 ts_manager.cc:194] Registered new tserver with Master: c32daee579a64f5c9b70a6bbdc1ee569 (127.14.147.193:36775)
I20260929 22:50:14.838941 14980 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.147.193@KRBTEST.COM'} at 127.14.147.193:59509
WARNING: no policy specified for kudu/127.14.147.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.147.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.147.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790722213368738-14927-0/krb5kdc/kudu_127.14.147.194.keytab.
Entry for principal kudu/127.14.147.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790722213368738-14927-0/krb5kdc/kudu_127.14.147.194.keytab.
I20260929 22:50:14.859571 14927 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhWOCLk/build/release/bin/kudu
/tmp/dist-test-taskhWOCLk/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790722213368738-14927-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790722213368738-14927-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790722213368738-14927-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790722213368738-14927-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.147.194:0
--local_ip_for_outbound_sockets=127.14.147.194
--webserver_interface=127.14.147.194
--webserver_port=0
--tserver_master_addrs=127.14.147.254:40593
--keytab_file=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790722213368738-14927-0/krb5kdc/kudu_127.14.147.194.keytab
--principal=kudu/127.14.147.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.147.212:34039
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790722213368738-14927-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790722213368738-14927-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790722213368738-14927-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 22:50:14.945007 15176 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 22:50:14.945225 15176 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 22:50:14.945250 15176 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 22:50:14.946444 15176 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 22:50:14.946486 15176 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 22:50:14.946514 15176 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.147.194
I20260929 22:50:14.948045 15176 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.147.212:34039
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790722213368738-14927-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790722213368738-14927-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790722213368738-14927-0/krb5kdc/kudu_127.14.147.194.keytab
--principal=kudu/127.14.147.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.147.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790722213368738-14927-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.14.147.194
--webserver_port=0
--tserver_master_addrs=127.14.147.254:40593
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15176
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.147.194
--log_dir=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790722213368738-14927-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 21:58:54 UTC on ccycloud.quasar-xguoti.root.comops.site
build id 4681390
I20260929 22:50:14.948256 15176 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 22:50:14.948448 15176 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 22:50:14.953233 15182 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 22:50:14.953233 15181 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 22:50:14.953233 15184 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 22:50:14.953523 15176 server_base.cc:1034] running on GCE node
I20260929 22:50:14.953706 15176 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 22:50:14.953974 15176 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 22:50:14.955170 15176 hybrid_clock.cc:648] HybridClock initialized: now 1790722214955156 us; error 31 us; skew 500 ppm
Sep 29 22:50:14 dist-test-slave-98mk krb5kdc[14934](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790722214, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.147.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 22:50:14.957908 15176 init.cc:377] Logged in from keytab as kudu/127.14.147.194@KRBTEST.COM (short username kudu)
I20260929 22:50:14.958372 15176 webserver.cc:460] Webserver started at http://127.14.147.194:44655/ using document root <none> and password file <none>
I20260929 22:50:14.958586 15176 fs_manager.cc:359] Metadata directory not provided
I20260929 22:50:14.958635 15176 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 22:50:14.958765 15176 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 22:50:14.959865 15176 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790722213368738-14927-0/minicluster-data/ts-1/data/instance:
uuid: "f235b09676f24e539773941391cf866b"
format_stamp: "Formatted at 2026-09-29 22:50:14 on dist-test-slave-98mk"
server_key: "8c6c4c0af6e3a1ff41ed7e4790cc765b"
server_key_iv: "432c73caa9513baebbca55772696318c"
server_key_version: "encryptionkey@0"
I20260929 22:50:14.960201 15176 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790722213368738-14927-0/minicluster-data/ts-1/wal/instance:
uuid: "f235b09676f24e539773941391cf866b"
format_stamp: "Formatted at 2026-09-29 22:50:14 on dist-test-slave-98mk"
server_key: "8c6c4c0af6e3a1ff41ed7e4790cc765b"
server_key_iv: "432c73caa9513baebbca55772696318c"
server_key_version: "encryptionkey@0"
I20260929 22:50:14.961537 15176 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260929 22:50:14.962289 15191 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 22:50:14.962466 15176 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260929 22:50:14.962548 15176 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790722213368738-14927-0/minicluster-data/ts-1/data,/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790722213368738-14927-0/minicluster-data/ts-1/wal
uuid: "f235b09676f24e539773941391cf866b"
format_stamp: "Formatted at 2026-09-29 22:50:14 on dist-test-slave-98mk"
server_key: "8c6c4c0af6e3a1ff41ed7e4790cc765b"
server_key_iv: "432c73caa9513baebbca55772696318c"
server_key_version: "encryptionkey@0"
I20260929 22:50:14.962610 15176 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790722213368738-14927-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790722213368738-14927-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790722213368738-14927-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 22:50:14.971690 15176 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 22:50:14.973568 15176 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 22:50:14.973707 15176 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 22:50:14.973932 15176 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 22:50:14.974227 15176 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 22:50:14.974258 15176 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 22:50:14.974287 15176 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 22:50:14.974303 15176 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 22:50:14.979987 15176 rpc_server.cc:307] RPC server started. Bound to: 127.14.147.194:43501
I20260929 22:50:14.980046 15304 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.147.194:43501 every 8 connection(s)
I20260929 22:50:14.980333 15176 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790722213368738-14927-0/minicluster-data/ts-1/data/info.pb
I20260929 22:50:14.984356 14927 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhWOCLk/build/release/bin/kudu as pid 15176
I20260929 22:50:14.984457 14927 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790722213368738-14927-0/minicluster-data/ts-1/wal/instance
I20260929 22:50:14.984627 14927 external_mini_cluster.cc:1428] Setting key a6466620dcc98bd56bc7546dbae65c71
Sep 29 22:50:14 dist-test-slave-98mk krb5kdc[14934](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790722214, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.147.194@KRBTEST.COM for kudu/127.14.147.254@KRBTEST.COM
I20260929 22:50:14.989701 15018 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:50:14.981003 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.147.194:44159 (local address 127.14.147.254:40593)
0929 22:50:14.981071 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0929 22:50:14.981073 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 22:50:14.981247 (+ 174us) server_negotiation.cc:407] Connection header received
0929 22:50:14.981753 (+ 506us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 22:50:14.981754 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 22:50:14.981767 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 22:50:14.981819 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0929 22:50:14.982760 (+ 941us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:50:14.983250 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:50:14.984282 (+ 1032us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:50:14.984484 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:50:14.984549 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 22:50:14.986890 (+ 2341us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 22:50:14.986892 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 22:50:14.986893 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 22:50:14.986900 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0929 22:50:14.988355 (+ 1455us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 22:50:14.988823 (+ 468us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 22:50:14.988825 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 22:50:14.988825 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 22:50:14.988885 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 22:50:14.989099 (+ 214us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 22:50:14.989100 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 22:50:14.989100 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 22:50:14.989223 (+ 123us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 22:50:14.989278 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0929 22:50:14.989530 (+ 252us) server_negotiation.cc:299] Negotiation successful
0929 22:50:14.989612 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260929 22:50:14.989888 15307 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:50:14.980945 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.147.254:40593 (local address 127.14.147.194:44159)
0929 22:50:14.981127 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0929 22:50:14.981168 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0929 22:50:14.981659 (+ 491us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 22:50:14.981889 (+ 230us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 22:50:14.981893 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 22:50:14.982463 (+ 570us) client_negotiation.cc:189] Negotiated authn=SASL
0929 22:50:14.982661 (+ 198us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 22:50:14.982666 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 22:50:14.983334 (+ 668us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 22:50:14.983335 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 22:50:14.984198 (+ 863us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 22:50:14.984201 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 22:50:14.984575 (+ 374us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 22:50:14.984577 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 22:50:14.984736 (+ 159us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 22:50:14.985286 (+ 550us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 22:50:14.985294 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0929 22:50:14.986800 (+ 1506us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 22:50:14.988450 (+ 1650us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 22:50:14.988456 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 22:50:14.988459 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 22:50:14.988750 (+ 291us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 22:50:14.988940 (+ 190us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 22:50:14.988940 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 22:50:14.988941 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 22:50:14.989044 (+ 103us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 22:50:14.989277 (+ 233us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 22:50:14.989284 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 22:50:14.989414 (+ 130us) client_negotiation.cc:769] Sending connection context
0929 22:50:14.989501 (+ 87us) client_negotiation.cc:240] Negotiation successful
0929 22:50:14.989678 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":140,"thread_start_us":95,"threads_started":1}
I20260929 22:50:14.990345 15305 heartbeater.cc:347] Connected to a master server at 127.14.147.254:40593
I20260929 22:50:14.990464 15305 heartbeater.cc:464] Registering TS with master...
I20260929 22:50:14.990665 15305 heartbeater.cc:511] Master 127.14.147.254:40593 requested a full tablet report, sending...
I20260929 22:50:14.991045 14980 ts_manager.cc:194] Registered new tserver with Master: f235b09676f24e539773941391cf866b (127.14.147.194:43501)
I20260929 22:50:14.991593 14980 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.147.194@KRBTEST.COM'} at 127.14.147.194:44159
WARNING: no policy specified for kudu/127.14.147.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.147.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.147.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790722213368738-14927-0/krb5kdc/kudu_127.14.147.195.keytab.
Entry for principal kudu/127.14.147.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790722213368738-14927-0/krb5kdc/kudu_127.14.147.195.keytab.
I20260929 22:50:15.007793 14927 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhWOCLk/build/release/bin/kudu
/tmp/dist-test-taskhWOCLk/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790722213368738-14927-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790722213368738-14927-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790722213368738-14927-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790722213368738-14927-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.147.195:0
--local_ip_for_outbound_sockets=127.14.147.195
--webserver_interface=127.14.147.195
--webserver_port=0
--tserver_master_addrs=127.14.147.254:40593
--keytab_file=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790722213368738-14927-0/krb5kdc/kudu_127.14.147.195.keytab
--principal=kudu/127.14.147.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.147.212:34039
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790722213368738-14927-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790722213368738-14927-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790722213368738-14927-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 22:50:15.088573 15310 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 22:50:15.088794 15310 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 22:50:15.088819 15310 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 22:50:15.090070 15310 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 22:50:15.090121 15310 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 22:50:15.090149 15310 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.147.195
I20260929 22:50:15.091750 15310 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.147.212:34039
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790722213368738-14927-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790722213368738-14927-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790722213368738-14927-0/krb5kdc/kudu_127.14.147.195.keytab
--principal=kudu/127.14.147.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.147.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790722213368738-14927-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.14.147.195
--webserver_port=0
--tserver_master_addrs=127.14.147.254:40593
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15310
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.147.195
--log_dir=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790722213368738-14927-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 21:58:54 UTC on ccycloud.quasar-xguoti.root.comops.site
build id 4681390
I20260929 22:50:15.092016 15310 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 22:50:15.092248 15310 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 22:50:15.097245 15318 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 22:50:15.097245 15315 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 22:50:15.097312 15316 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 22:50:15.097620 15310 server_base.cc:1034] running on GCE node
I20260929 22:50:15.097810 15310 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 22:50:15.098081 15310 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 22:50:15.099236 15310 hybrid_clock.cc:648] HybridClock initialized: now 1790722215099217 us; error 30 us; skew 500 ppm
Sep 29 22:50:15 dist-test-slave-98mk krb5kdc[14934](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790722215, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.147.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 22:50:15.101783 15310 init.cc:377] Logged in from keytab as kudu/127.14.147.195@KRBTEST.COM (short username kudu)
I20260929 22:50:15.102267 15310 webserver.cc:460] Webserver started at http://127.14.147.195:39049/ using document root <none> and password file <none>
I20260929 22:50:15.102486 15310 fs_manager.cc:359] Metadata directory not provided
I20260929 22:50:15.102535 15310 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 22:50:15.102658 15310 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 22:50:15.103786 15310 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790722213368738-14927-0/minicluster-data/ts-2/data/instance:
uuid: "a34f6519f83b48328b195278bb682b9a"
format_stamp: "Formatted at 2026-09-29 22:50:15 on dist-test-slave-98mk"
server_key: "d7e3a2412c47441c157b2b1f80533a5d"
server_key_iv: "65b2786b8585ef1a244724abbe244a91"
server_key_version: "encryptionkey@0"
I20260929 22:50:15.104132 15310 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790722213368738-14927-0/minicluster-data/ts-2/wal/instance:
uuid: "a34f6519f83b48328b195278bb682b9a"
format_stamp: "Formatted at 2026-09-29 22:50:15 on dist-test-slave-98mk"
server_key: "d7e3a2412c47441c157b2b1f80533a5d"
server_key_iv: "65b2786b8585ef1a244724abbe244a91"
server_key_version: "encryptionkey@0"
I20260929 22:50:15.105461 15310 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260929 22:50:15.106554 15325 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 22:50:15.106864 15310 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260929 22:50:15.106978 15310 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790722213368738-14927-0/minicluster-data/ts-2/data,/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790722213368738-14927-0/minicluster-data/ts-2/wal
uuid: "a34f6519f83b48328b195278bb682b9a"
format_stamp: "Formatted at 2026-09-29 22:50:15 on dist-test-slave-98mk"
server_key: "d7e3a2412c47441c157b2b1f80533a5d"
server_key_iv: "65b2786b8585ef1a244724abbe244a91"
server_key_version: "encryptionkey@0"
I20260929 22:50:15.107060 15310 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790722213368738-14927-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790722213368738-14927-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790722213368738-14927-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 22:50:15.122143 15310 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 22:50:15.124432 15310 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 22:50:15.124549 15310 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 22:50:15.124774 15310 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 22:50:15.125088 15310 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 22:50:15.125123 15310 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 22:50:15.125144 15310 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 22:50:15.125157 15310 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 22:50:15.130707 15310 rpc_server.cc:307] RPC server started. Bound to: 127.14.147.195:36127
I20260929 22:50:15.130772 15438 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.147.195:36127 every 8 connection(s)
I20260929 22:50:15.131071 15310 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790722213368738-14927-0/minicluster-data/ts-2/data/info.pb
I20260929 22:50:15.132817 14927 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhWOCLk/build/release/bin/kudu as pid 15310
I20260929 22:50:15.133064 14927 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790722213368738-14927-0/minicluster-data/ts-2/wal/instance
I20260929 22:50:15.133482 14927 external_mini_cluster.cc:1428] Setting key fdc9886b066d6e363f510135aa791077
Sep 29 22:50:15 dist-test-slave-98mk krb5kdc[14934](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790722215, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.147.195@KRBTEST.COM for kudu/127.14.147.254@KRBTEST.COM
I20260929 22:50:15.141428 15018 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:50:15.131818 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.147.195:35151 (local address 127.14.147.254:40593)
0929 22:50:15.131881 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0929 22:50:15.131884 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 22:50:15.132095 (+ 211us) server_negotiation.cc:407] Connection header received
0929 22:50:15.132723 (+ 628us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 22:50:15.132726 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 22:50:15.132753 (+ 27us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 22:50:15.132998 (+ 245us) server_negotiation.cc:226] Negotiated authn=SASL
0929 22:50:15.134377 (+ 1379us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:50:15.135019 (+ 642us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:50:15.136329 (+ 1310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:50:15.136531 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:50:15.136591 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 22:50:15.138655 (+ 2064us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 22:50:15.138656 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 22:50:15.138656 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 22:50:15.138663 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0929 22:50:15.139958 (+ 1295us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 22:50:15.140377 (+ 419us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 22:50:15.140378 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 22:50:15.140378 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 22:50:15.140434 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 22:50:15.140634 (+ 200us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 22:50:15.140634 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 22:50:15.140634 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 22:50:15.140741 (+ 107us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 22:50:15.140790 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0929 22:50:15.141277 (+ 487us) server_negotiation.cc:299] Negotiation successful
0929 22:50:15.141348 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260929 22:50:15.141676 15441 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:50:15.131784 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.147.254:40593 (local address 127.14.147.195:35151)
0929 22:50:15.131977 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0929 22:50:15.132016 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0929 22:50:15.132506 (+ 490us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 22:50:15.133289 (+ 783us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 22:50:15.133296 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 22:50:15.133970 (+ 674us) client_negotiation.cc:189] Negotiated authn=SASL
0929 22:50:15.134264 (+ 294us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 22:50:15.134270 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 22:50:15.135144 (+ 874us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 22:50:15.135146 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 22:50:15.136241 (+ 1095us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 22:50:15.136247 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 22:50:15.136602 (+ 355us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 22:50:15.136603 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 22:50:15.136718 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 22:50:15.137202 (+ 484us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 22:50:15.137211 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0929 22:50:15.138547 (+ 1336us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 22:50:15.140042 (+ 1495us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 22:50:15.140049 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 22:50:15.140051 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 22:50:15.140282 (+ 231us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 22:50:15.140491 (+ 209us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 22:50:15.140492 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 22:50:15.140492 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 22:50:15.140591 (+ 99us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 22:50:15.140840 (+ 249us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 22:50:15.140849 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 22:50:15.141053 (+ 204us) client_negotiation.cc:769] Sending connection context
0929 22:50:15.141274 (+ 221us) client_negotiation.cc:240] Negotiation successful
0929 22:50:15.141452 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":146,"thread_start_us":90,"threads_started":1}
I20260929 22:50:15.142217 15439 heartbeater.cc:347] Connected to a master server at 127.14.147.254:40593
I20260929 22:50:15.142316 15439 heartbeater.cc:464] Registering TS with master...
I20260929 22:50:15.142506 15439 heartbeater.cc:511] Master 127.14.147.254:40593 requested a full tablet report, sending...
I20260929 22:50:15.142854 14980 ts_manager.cc:194] Registered new tserver with Master: a34f6519f83b48328b195278bb682b9a (127.14.147.195:36127)
I20260929 22:50:15.143200 14980 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.147.195@KRBTEST.COM'} at 127.14.147.195:35151
I20260929 22:50:15.145414 14927 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 29 22:50:15 dist-test-slave-98mk krb5kdc[14934](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790722215, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 29 22:50:15 dist-test-slave-98mk krb5kdc[14934](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790722215, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.14.147.193@KRBTEST.COM
I20260929 22:50:15.167927 15446 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:50:15.160553 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47002 (local address 127.14.147.193:36775)
0929 22:50:15.160715 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0929 22:50:15.160721 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 22:50:15.160732 (+ 11us) server_negotiation.cc:407] Connection header received
0929 22:50:15.160759 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 22:50:15.160761 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 22:50:15.160775 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 22:50:15.160840 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0929 22:50:15.161406 (+ 566us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:50:15.162147 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:50:15.162864 (+ 717us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:50:15.163195 (+ 331us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:50:15.163255 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 22:50:15.165415 (+ 2160us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 22:50:15.165427 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 22:50:15.165427 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 22:50:15.165435 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0929 22:50:15.166791 (+ 1356us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 22:50:15.167107 (+ 316us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 22:50:15.167110 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 22:50:15.167111 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 22:50:15.167176 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 22:50:15.167339 (+ 163us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 22:50:15.167340 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 22:50:15.167340 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 22:50:15.167521 (+ 181us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 22:50:15.167661 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0929 22:50:15.167738 (+ 77us) server_negotiation.cc:299] Negotiation successful
0929 22:50:15.167819 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":123,"thread_start_us":68,"threads_started":1}
W20260929 22:50:15.168309 15085 server_base.cc:1116] Unauthorized access attempt to method kudu.tserver.TabletServerService.ListTablets from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:47002
Sep 29 22:50:15 dist-test-slave-98mk krb5kdc[14934](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790722215, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 29 22:50:15 dist-test-slave-98mk krb5kdc[14934](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790722215, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.147.193@KRBTEST.COM
I20260929 22:50:15.189990 15446 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:50:15.182957 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47018 (local address 127.14.147.193:36775)
0929 22:50:15.183059 (+ 102us) server_negotiation.cc:206] Beginning negotiation
0929 22:50:15.183061 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 22:50:15.183121 (+ 60us) server_negotiation.cc:407] Connection header received
0929 22:50:15.183207 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 22:50:15.183208 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 22:50:15.183219 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 22:50:15.183262 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0929 22:50:15.183927 (+ 665us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:50:15.184428 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:50:15.185189 (+ 761us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:50:15.185396 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:50:15.185470 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 22:50:15.187712 (+ 2242us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 22:50:15.187714 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 22:50:15.187714 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 22:50:15.187727 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0929 22:50:15.188933 (+ 1206us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 22:50:15.189270 (+ 337us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 22:50:15.189272 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 22:50:15.189272 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 22:50:15.189324 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 22:50:15.189488 (+ 164us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 22:50:15.189489 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 22:50:15.189489 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 22:50:15.189595 (+ 106us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 22:50:15.189708 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0929 22:50:15.189789 (+ 81us) server_negotiation.cc:299] Negotiation successful
0929 22:50:15.189877 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260929 22:50:15.190791 14927 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhWOCLk/build/release/bin/kudu with pid 15042
I20260929 22:50:15.195113 14927 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhWOCLk/build/release/bin/kudu with pid 15176
I20260929 22:50:15.199250 14927 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhWOCLk/build/release/bin/kudu with pid 15310
I20260929 22:50:15.203189 14927 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhWOCLk/build/release/bin/kudu with pid 14947
2026-09-29T22:50:15Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnListTablets (1841 ms)
[ RUN ] SecurityITest.TestAuthorizationOnChecksum
Loading random data
Initializing database '/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790722213368738-14927-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 22:50:15 dist-test-slave-98mk krb5kdc[15453](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 22:50:15 dist-test-slave-98mk krb5kdc[15453](info): set up 2 sockets
Sep 29 22:50:15 dist-test-slave-98mk krb5kdc[15453](info): commencing operation
krb5kdc: starting...
W20260929 22:50:16.151561 14927 mini_kdc.cc:121] Time spent starting KDC: real 0.929s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790722213368738-14927-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790722213368738-14927-0/krb5kdc/test-user.keytab.
Sep 29 22:50:16 dist-test-slave-98mk krb5kdc[15453](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790722216, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T22:50:16Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T22:50:16Z Disabled control of system clock
WARNING: no policy specified for kudu/127.14.147.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.147.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.147.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790722213368738-14927-0/krb5kdc/kudu_127.14.147.254.keytab.
Entry for principal kudu/127.14.147.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790722213368738-14927-0/krb5kdc/kudu_127.14.147.254.keytab.
I20260929 22:50:16.268091 14927 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhWOCLk/build/release/bin/kudu
/tmp/dist-test-taskhWOCLk/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790722213368738-14927-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790722213368738-14927-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790722213368738-14927-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790722213368738-14927-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.147.254:35003
--webserver_interface=127.14.147.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790722213368738-14927-0/krb5kdc/kudu_127.14.147.254.keytab
--principal=kudu/127.14.147.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.147.212:34039
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.147.254:35003
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790722213368738-14927-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790722213368738-14927-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790722213368738-14927-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 22:50:16.349890 15466 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 22:50:16.350152 15466 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 22:50:16.350184 15466 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 22:50:16.351406 15466 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 22:50:16.351495 15466 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 22:50:16.351514 15466 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 22:50:16.351531 15466 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 22:50:16.351545 15466 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 22:50:16.353156 15466 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.147.212:34039
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790722213368738-14927-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790722213368738-14927-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.14.147.254:35003
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790722213368738-14927-0/krb5kdc/kudu_127.14.147.254.keytab
--principal=kudu/127.14.147.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.147.254:35003
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790722213368738-14927-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.14.147.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15466
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790722213368738-14927-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 21:58:54 UTC on ccycloud.quasar-xguoti.root.comops.site
build id 4681390
I20260929 22:50:16.353425 15466 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 22:50:16.353652 15466 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 22:50:16.358444 15471 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 22:50:16.358505 15466 server_base.cc:1034] running on GCE node
W20260929 22:50:16.358440 15474 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 22:50:16.358438 15472 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 22:50:16.358898 15466 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 22:50:16.359198 15466 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 22:50:16.360348 15466 hybrid_clock.cc:648] HybridClock initialized: now 1790722216360345 us; error 44 us; skew 500 ppm
Sep 29 22:50:16 dist-test-slave-98mk krb5kdc[15453](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790722216, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.147.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 22:50:16.362882 15466 init.cc:377] Logged in from keytab as kudu/127.14.147.254@KRBTEST.COM (short username kudu)
I20260929 22:50:16.363365 15466 webserver.cc:460] Webserver started at http://127.14.147.254:38935/ using document root <none> and password file <none>
I20260929 22:50:16.363628 15466 fs_manager.cc:359] Metadata directory not provided
I20260929 22:50:16.363680 15466 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 22:50:16.363809 15466 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 22:50:16.364924 15466 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790722213368738-14927-0/minicluster-data/master-0/data/instance:
uuid: "87b861b67697436c80de741f2771f2d8"
format_stamp: "Formatted at 2026-09-29 22:50:16 on dist-test-slave-98mk"
server_key: "55bc13ef6e9b347dc2b48855e39a7cd1"
server_key_iv: "304ba3b13a69387aa63c8145396d7e47"
server_key_version: "encryptionkey@0"
I20260929 22:50:16.365345 15466 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790722213368738-14927-0/minicluster-data/master-0/wal/instance:
uuid: "87b861b67697436c80de741f2771f2d8"
format_stamp: "Formatted at 2026-09-29 22:50:16 on dist-test-slave-98mk"
server_key: "55bc13ef6e9b347dc2b48855e39a7cd1"
server_key_iv: "304ba3b13a69387aa63c8145396d7e47"
server_key_version: "encryptionkey@0"
I20260929 22:50:16.366673 15466 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260929 22:50:16.367725 15481 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 22:50:16.367995 15466 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260929 22:50:16.368110 15466 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790722213368738-14927-0/minicluster-data/master-0/data,/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790722213368738-14927-0/minicluster-data/master-0/wal
uuid: "87b861b67697436c80de741f2771f2d8"
format_stamp: "Formatted at 2026-09-29 22:50:16 on dist-test-slave-98mk"
server_key: "55bc13ef6e9b347dc2b48855e39a7cd1"
server_key_iv: "304ba3b13a69387aa63c8145396d7e47"
server_key_version: "encryptionkey@0"
I20260929 22:50:16.368189 15466 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790722213368738-14927-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790722213368738-14927-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790722213368738-14927-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 22:50:16.393483 15466 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 22:50:16.395785 15466 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 22:50:16.395946 15466 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 22:50:16.399925 15466 rpc_server.cc:307] RPC server started. Bound to: 127.14.147.254:35003
I20260929 22:50:16.399963 15533 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.147.254:35003 every 8 connection(s)
I20260929 22:50:16.400313 15466 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790722213368738-14927-0/minicluster-data/master-0/data/info.pb
I20260929 22:50:16.401176 15534 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 22:50:16.403072 14927 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhWOCLk/build/release/bin/kudu as pid 15466
I20260929 22:50:16.403158 14927 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790722213368738-14927-0/minicluster-data/master-0/wal/instance
I20260929 22:50:16.403312 14927 external_mini_cluster.cc:1428] Setting key 7f9639c544b11e57e89ea27fc9b056fb
I20260929 22:50:16.403642 15534 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 87b861b67697436c80de741f2771f2d8: Bootstrap starting.
I20260929 22:50:16.404296 15534 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 87b861b67697436c80de741f2771f2d8: Neither blocks nor log segments found. Creating new log.
I20260929 22:50:16.404561 15534 log.cc:824] T 00000000000000000000000000000000 P 87b861b67697436c80de741f2771f2d8: Log is configured to *not* fsync() on all Append() calls
I20260929 22:50:16.405627 15534 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 87b861b67697436c80de741f2771f2d8: No bootstrap required, opened a new log
I20260929 22:50:16.407562 15534 raft_consensus.cc:348] T 00000000000000000000000000000000 P 87b861b67697436c80de741f2771f2d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "87b861b67697436c80de741f2771f2d8" member_type: VOTER last_known_addr { host: "127.14.147.254" port: 35003 } }
I20260929 22:50:16.407718 15534 raft_consensus.cc:374] T 00000000000000000000000000000000 P 87b861b67697436c80de741f2771f2d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 22:50:16.407748 15534 raft_consensus.cc:729] T 00000000000000000000000000000000 P 87b861b67697436c80de741f2771f2d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 87b861b67697436c80de741f2771f2d8, State: Initialized, Role: FOLLOWER
I20260929 22:50:16.407881 15534 consensus_queue.cc:260] T 00000000000000000000000000000000 P 87b861b67697436c80de741f2771f2d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "87b861b67697436c80de741f2771f2d8" member_type: VOTER last_known_addr { host: "127.14.147.254" port: 35003 } }
I20260929 22:50:16.407958 15534 raft_consensus.cc:388] T 00000000000000000000000000000000 P 87b861b67697436c80de741f2771f2d8 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 22:50:16.407994 15534 raft_consensus.cc:482] T 00000000000000000000000000000000 P 87b861b67697436c80de741f2771f2d8 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 22:50:16.408034 15534 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 87b861b67697436c80de741f2771f2d8 [term 0 FOLLOWER]: Advancing to term 1
I20260929 22:50:16.408695 15534 raft_consensus.cc:504] T 00000000000000000000000000000000 P 87b861b67697436c80de741f2771f2d8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "87b861b67697436c80de741f2771f2d8" member_type: VOTER last_known_addr { host: "127.14.147.254" port: 35003 } }
I20260929 22:50:16.408798 15534 leader_election.cc:302] T 00000000000000000000000000000000 P 87b861b67697436c80de741f2771f2d8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 87b861b67697436c80de741f2771f2d8; no voters:
I20260929 22:50:16.408973 15534 leader_election.cc:288] T 00000000000000000000000000000000 P 87b861b67697436c80de741f2771f2d8 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 22:50:16.409017 15539 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 87b861b67697436c80de741f2771f2d8 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 22:50:16.409147 15539 raft_consensus.cc:686] T 00000000000000000000000000000000 P 87b861b67697436c80de741f2771f2d8 [term 1 LEADER]: Becoming Leader. State: Replica: 87b861b67697436c80de741f2771f2d8, State: Running, Role: LEADER
I20260929 22:50:16.409226 15539 consensus_queue.cc:237] T 00000000000000000000000000000000 P 87b861b67697436c80de741f2771f2d8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "87b861b67697436c80de741f2771f2d8" member_type: VOTER last_known_addr { host: "127.14.147.254" port: 35003 } }
I20260929 22:50:16.409252 15534 sys_catalog.cc:564] T 00000000000000000000000000000000 P 87b861b67697436c80de741f2771f2d8 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 22:50:16.409577 15539 sys_catalog.cc:455] T 00000000000000000000000000000000 P 87b861b67697436c80de741f2771f2d8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 87b861b67697436c80de741f2771f2d8. Latest consensus state: current_term: 1 leader_uuid: "87b861b67697436c80de741f2771f2d8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "87b861b67697436c80de741f2771f2d8" member_type: VOTER last_known_addr { host: "127.14.147.254" port: 35003 } } }
I20260929 22:50:16.409663 15539 sys_catalog.cc:458] T 00000000000000000000000000000000 P 87b861b67697436c80de741f2771f2d8 [sys.catalog]: This master's current role is: LEADER
Sep 29 22:50:16 dist-test-slave-98mk krb5kdc[15453](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790722216, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.147.254@KRBTEST.COM
I20260929 22:50:16.409591 15541 sys_catalog.cc:455] T 00000000000000000000000000000000 P 87b861b67697436c80de741f2771f2d8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "87b861b67697436c80de741f2771f2d8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "87b861b67697436c80de741f2771f2d8" member_type: VOTER last_known_addr { host: "127.14.147.254" port: 35003 } } }
I20260929 22:50:16.409878 15541 sys_catalog.cc:458] T 00000000000000000000000000000000 P 87b861b67697436c80de741f2771f2d8 [sys.catalog]: This master's current role is: LEADER
I20260929 22:50:16.410012 15545 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 22:50:16.410537 15545 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 22:50:16.413887 15537 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:50:16.403819 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49374 (local address 127.14.147.254:35003)
0929 22:50:16.404011 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0929 22:50:16.404019 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 22:50:16.404046 (+ 27us) server_negotiation.cc:407] Connection header received
0929 22:50:16.404532 (+ 486us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 22:50:16.404539 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 22:50:16.404591 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 22:50:16.404744 (+ 153us) server_negotiation.cc:226] Negotiated authn=SASL
0929 22:50:16.405472 (+ 728us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:50:16.406416 (+ 944us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:50:16.407178 (+ 762us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:50:16.407645 (+ 467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:50:16.407742 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 22:50:16.410330 (+ 2588us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 22:50:16.410335 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 22:50:16.410335 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 22:50:16.410349 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0929 22:50:16.412285 (+ 1936us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 22:50:16.412658 (+ 373us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 22:50:16.412661 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 22:50:16.412662 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 22:50:16.412774 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 22:50:16.412937 (+ 163us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 22:50:16.412938 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 22:50:16.412938 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 22:50:16.413263 (+ 325us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 22:50:16.413403 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0929 22:50:16.413519 (+ 116us) server_negotiation.cc:299] Negotiation successful
0929 22:50:16.413665 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":140,"thread_start_us":88,"threads_started":1}
I20260929 22:50:16.414079 15545 catalog_manager.cc:1347] Generated new cluster ID: d07be79a284743c688440cb70c9b04d5
I20260929 22:50:16.414113 15545 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 22:50:16.426190 15545 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 22:50:16.426939 15545 catalog_manager.cc:1504] Loading token signing keys...
I20260929 22:50:16.441638 15545 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 87b861b67697436c80de741f2771f2d8: Generated new TSK 0
I20260929 22:50:16.441864 15545 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.14.147.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.147.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.147.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790722213368738-14927-0/krb5kdc/kudu_127.14.147.193.keytab.
Entry for principal kudu/127.14.147.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790722213368738-14927-0/krb5kdc/kudu_127.14.147.193.keytab.
I20260929 22:50:16.469717 14927 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhWOCLk/build/release/bin/kudu
/tmp/dist-test-taskhWOCLk/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790722213368738-14927-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790722213368738-14927-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790722213368738-14927-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790722213368738-14927-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.147.193:0
--local_ip_for_outbound_sockets=127.14.147.193
--webserver_interface=127.14.147.193
--webserver_port=0
--tserver_master_addrs=127.14.147.254:35003
--keytab_file=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790722213368738-14927-0/krb5kdc/kudu_127.14.147.193.keytab
--principal=kudu/127.14.147.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.147.212:34039
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790722213368738-14927-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790722213368738-14927-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790722213368738-14927-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 22:50:16.555365 15560 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 22:50:16.555657 15560 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 22:50:16.555693 15560 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 22:50:16.556957 15560 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 22:50:16.557001 15560 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 22:50:16.557052 15560 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.147.193
I20260929 22:50:16.559266 15560 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.147.212:34039
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790722213368738-14927-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790722213368738-14927-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790722213368738-14927-0/krb5kdc/kudu_127.14.147.193.keytab
--principal=kudu/127.14.147.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.147.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790722213368738-14927-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.14.147.193
--webserver_port=0
--tserver_master_addrs=127.14.147.254:35003
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15560
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.147.193
--log_dir=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790722213368738-14927-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 21:58:54 UTC on ccycloud.quasar-xguoti.root.comops.site
build id 4681390
I20260929 22:50:16.559537 15560 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 22:50:16.559752 15560 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 22:50:16.564560 15565 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 22:50:16.564610 15560 server_base.cc:1034] running on GCE node
W20260929 22:50:16.564560 15566 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 22:50:16.564553 15568 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 22:50:16.564890 15560 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 22:50:16.565105 15560 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 22:50:16.566250 15560 hybrid_clock.cc:648] HybridClock initialized: now 1790722216566229 us; error 31 us; skew 500 ppm
Sep 29 22:50:16 dist-test-slave-98mk krb5kdc[15453](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790722216, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.147.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 22:50:16.568781 15560 init.cc:377] Logged in from keytab as kudu/127.14.147.193@KRBTEST.COM (short username kudu)
I20260929 22:50:16.569201 15560 webserver.cc:460] Webserver started at http://127.14.147.193:43277/ using document root <none> and password file <none>
I20260929 22:50:16.569377 15560 fs_manager.cc:359] Metadata directory not provided
I20260929 22:50:16.569418 15560 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 22:50:16.569515 15560 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 22:50:16.570568 15560 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790722213368738-14927-0/minicluster-data/ts-0/data/instance:
uuid: "461a4dc5d5fa4b9f8cac432d45aa8913"
format_stamp: "Formatted at 2026-09-29 22:50:16 on dist-test-slave-98mk"
server_key: "a72dbedb2ebc06af0068dc1558e2e3c0"
server_key_iv: "5edb3a4a7a18703425a6ebf7ad9901e2"
server_key_version: "encryptionkey@0"
I20260929 22:50:16.570874 15560 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790722213368738-14927-0/minicluster-data/ts-0/wal/instance:
uuid: "461a4dc5d5fa4b9f8cac432d45aa8913"
format_stamp: "Formatted at 2026-09-29 22:50:16 on dist-test-slave-98mk"
server_key: "a72dbedb2ebc06af0068dc1558e2e3c0"
server_key_iv: "5edb3a4a7a18703425a6ebf7ad9901e2"
server_key_version: "encryptionkey@0"
I20260929 22:50:16.572381 15560 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260929 22:50:16.573138 15575 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 22:50:16.573338 15560 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260929 22:50:16.573417 15560 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790722213368738-14927-0/minicluster-data/ts-0/data,/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790722213368738-14927-0/minicluster-data/ts-0/wal
uuid: "461a4dc5d5fa4b9f8cac432d45aa8913"
format_stamp: "Formatted at 2026-09-29 22:50:16 on dist-test-slave-98mk"
server_key: "a72dbedb2ebc06af0068dc1558e2e3c0"
server_key_iv: "5edb3a4a7a18703425a6ebf7ad9901e2"
server_key_version: "encryptionkey@0"
I20260929 22:50:16.573482 15560 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790722213368738-14927-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790722213368738-14927-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790722213368738-14927-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 22:50:16.595772 15560 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 22:50:16.597947 15560 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 22:50:16.598065 15560 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 22:50:16.598260 15560 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 22:50:16.598519 15560 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 22:50:16.598549 15560 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 22:50:16.598572 15560 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 22:50:16.598585 15560 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 22:50:16.604460 15560 rpc_server.cc:307] RPC server started. Bound to: 127.14.147.193:39605
I20260929 22:50:16.604532 15688 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.147.193:39605 every 8 connection(s)
I20260929 22:50:16.604823 15560 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790722213368738-14927-0/minicluster-data/ts-0/data/info.pb
I20260929 22:50:16.605283 14927 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhWOCLk/build/release/bin/kudu as pid 15560
I20260929 22:50:16.605376 14927 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790722213368738-14927-0/minicluster-data/ts-0/wal/instance
I20260929 22:50:16.605545 14927 external_mini_cluster.cc:1428] Setting key 8d0794f104962c852a42f63f72c8c9ea
Sep 29 22:50:16 dist-test-slave-98mk krb5kdc[15453](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790722216, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.147.193@KRBTEST.COM for kudu/127.14.147.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.14.147.194@KRBTEST.COM; defaulting to no policy
I20260929 22:50:16.615346 15537 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:50:16.605531 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.147.193:34675 (local address 127.14.147.254:35003)
0929 22:50:16.605781 (+ 250us) server_negotiation.cc:206] Beginning negotiation
0929 22:50:16.605783 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 22:50:16.605831 (+ 48us) server_negotiation.cc:407] Connection header received
0929 22:50:16.606825 (+ 994us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 22:50:16.606826 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 22:50:16.606838 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 22:50:16.606891 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0929 22:50:16.607930 (+ 1039us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:50:16.608641 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:50:16.609782 (+ 1141us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:50:16.610119 (+ 337us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:50:16.610190 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 22:50:16.612684 (+ 2494us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 22:50:16.612686 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 22:50:16.612686 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 22:50:16.612693 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0929 22:50:16.614033 (+ 1340us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 22:50:16.614406 (+ 373us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 22:50:16.614407 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 22:50:16.614407 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 22:50:16.614463 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 22:50:16.614701 (+ 238us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 22:50:16.614702 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 22:50:16.614702 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 22:50:16.614821 (+ 119us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 22:50:16.614878 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0929 22:50:16.615159 (+ 281us) server_negotiation.cc:299] Negotiation successful
0929 22:50:16.615251 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":220}
I20260929 22:50:16.615551 15691 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:50:16.605539 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.147.254:35003 (local address 127.14.147.193:34675)
0929 22:50:16.605708 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0929 22:50:16.605761 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0929 22:50:16.606720 (+ 959us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 22:50:16.607058 (+ 338us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 22:50:16.607064 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 22:50:16.607601 (+ 537us) client_negotiation.cc:189] Negotiated authn=SASL
0929 22:50:16.607826 (+ 225us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 22:50:16.607834 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 22:50:16.608741 (+ 907us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 22:50:16.608742 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 22:50:16.609696 (+ 954us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 22:50:16.609699 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 22:50:16.610200 (+ 501us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 22:50:16.610201 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 22:50:16.610313 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 22:50:16.610911 (+ 598us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 22:50:16.610921 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0929 22:50:16.612589 (+ 1668us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 22:50:16.614106 (+ 1517us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 22:50:16.614114 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 22:50:16.614116 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 22:50:16.614339 (+ 223us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 22:50:16.614526 (+ 187us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 22:50:16.614527 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 22:50:16.614527 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 22:50:16.614639 (+ 112us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 22:50:16.614882 (+ 243us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 22:50:16.614888 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 22:50:16.615034 (+ 146us) client_negotiation.cc:769] Sending connection context
0929 22:50:16.615129 (+ 95us) client_negotiation.cc:240] Negotiation successful
0929 22:50:16.615296 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":86,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.147.194@KRBTEST.COM" created.
I20260929 22:50:16.616027 15689 heartbeater.cc:347] Connected to a master server at 127.14.147.254:35003
I20260929 22:50:16.616140 15689 heartbeater.cc:464] Registering TS with master...
I20260929 22:50:16.616319 15689 heartbeater.cc:511] Master 127.14.147.254:35003 requested a full tablet report, sending...
I20260929 22:50:16.616845 15498 ts_manager.cc:194] Registered new tserver with Master: 461a4dc5d5fa4b9f8cac432d45aa8913 (127.14.147.193:39605)
I20260929 22:50:16.617465 15498 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.147.193@KRBTEST.COM'} at 127.14.147.193:34675
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.147.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790722213368738-14927-0/krb5kdc/kudu_127.14.147.194.keytab.
Entry for principal kudu/127.14.147.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790722213368738-14927-0/krb5kdc/kudu_127.14.147.194.keytab.
I20260929 22:50:16.629333 14927 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhWOCLk/build/release/bin/kudu
/tmp/dist-test-taskhWOCLk/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790722213368738-14927-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790722213368738-14927-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790722213368738-14927-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790722213368738-14927-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.147.194:0
--local_ip_for_outbound_sockets=127.14.147.194
--webserver_interface=127.14.147.194
--webserver_port=0
--tserver_master_addrs=127.14.147.254:35003
--keytab_file=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790722213368738-14927-0/krb5kdc/kudu_127.14.147.194.keytab
--principal=kudu/127.14.147.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.147.212:34039
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790722213368738-14927-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790722213368738-14927-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790722213368738-14927-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 22:50:16.710695 15694 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 22:50:16.710922 15694 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 22:50:16.710959 15694 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 22:50:16.712293 15694 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 22:50:16.712343 15694 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 22:50:16.712373 15694 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.147.194
I20260929 22:50:16.713894 15694 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.147.212:34039
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790722213368738-14927-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790722213368738-14927-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790722213368738-14927-0/krb5kdc/kudu_127.14.147.194.keytab
--principal=kudu/127.14.147.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.147.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790722213368738-14927-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.14.147.194
--webserver_port=0
--tserver_master_addrs=127.14.147.254:35003
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15694
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.147.194
--log_dir=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790722213368738-14927-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 21:58:54 UTC on ccycloud.quasar-xguoti.root.comops.site
build id 4681390
I20260929 22:50:16.714103 15694 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 22:50:16.714290 15694 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 22:50:16.719081 15699 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 22:50:16.719062 15702 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 22:50:16.719272 15694 server_base.cc:1034] running on GCE node
W20260929 22:50:16.719067 15700 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 22:50:16.719534 15694 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 22:50:16.719746 15694 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 22:50:16.720908 15694 hybrid_clock.cc:648] HybridClock initialized: now 1790722216720881 us; error 44 us; skew 500 ppm
Sep 29 22:50:16 dist-test-slave-98mk krb5kdc[15453](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790722216, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.147.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 22:50:16.723385 15694 init.cc:377] Logged in from keytab as kudu/127.14.147.194@KRBTEST.COM (short username kudu)
I20260929 22:50:16.723860 15694 webserver.cc:460] Webserver started at http://127.14.147.194:34287/ using document root <none> and password file <none>
I20260929 22:50:16.724058 15694 fs_manager.cc:359] Metadata directory not provided
I20260929 22:50:16.724107 15694 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 22:50:16.724222 15694 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 22:50:16.725258 15694 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790722213368738-14927-0/minicluster-data/ts-1/data/instance:
uuid: "fa42e58470984c3c90abc119f317b887"
format_stamp: "Formatted at 2026-09-29 22:50:16 on dist-test-slave-98mk"
server_key: "6aaefab95e49b25a253eb626d14a98dd"
server_key_iv: "ae4aa1c0651c47072adfd0f5b58cd4c1"
server_key_version: "encryptionkey@0"
I20260929 22:50:16.725582 15694 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790722213368738-14927-0/minicluster-data/ts-1/wal/instance:
uuid: "fa42e58470984c3c90abc119f317b887"
format_stamp: "Formatted at 2026-09-29 22:50:16 on dist-test-slave-98mk"
server_key: "6aaefab95e49b25a253eb626d14a98dd"
server_key_iv: "ae4aa1c0651c47072adfd0f5b58cd4c1"
server_key_version: "encryptionkey@0"
I20260929 22:50:16.726828 15694 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260929 22:50:16.727480 15709 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 22:50:16.727626 15694 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260929 22:50:16.727707 15694 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790722213368738-14927-0/minicluster-data/ts-1/data,/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790722213368738-14927-0/minicluster-data/ts-1/wal
uuid: "fa42e58470984c3c90abc119f317b887"
format_stamp: "Formatted at 2026-09-29 22:50:16 on dist-test-slave-98mk"
server_key: "6aaefab95e49b25a253eb626d14a98dd"
server_key_iv: "ae4aa1c0651c47072adfd0f5b58cd4c1"
server_key_version: "encryptionkey@0"
I20260929 22:50:16.727769 15694 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790722213368738-14927-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790722213368738-14927-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790722213368738-14927-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 22:50:16.752923 15694 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 22:50:16.755116 15694 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 22:50:16.755259 15694 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 22:50:16.755520 15694 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 22:50:16.755831 15694 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 22:50:16.755863 15694 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 22:50:16.755893 15694 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 22:50:16.755910 15694 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 22:50:16.761445 15694 rpc_server.cc:307] RPC server started. Bound to: 127.14.147.194:43245
I20260929 22:50:16.761504 15822 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.147.194:43245 every 8 connection(s)
I20260929 22:50:16.761793 15694 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790722213368738-14927-0/minicluster-data/ts-1/data/info.pb
I20260929 22:50:16.764346 14927 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhWOCLk/build/release/bin/kudu as pid 15694
I20260929 22:50:16.764436 14927 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790722213368738-14927-0/minicluster-data/ts-1/wal/instance
I20260929 22:50:16.764617 14927 external_mini_cluster.cc:1428] Setting key 4084d093746398700f149c0cfb60b2f7
Sep 29 22:50:16 dist-test-slave-98mk krb5kdc[15453](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790722216, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.147.194@KRBTEST.COM for kudu/127.14.147.254@KRBTEST.COM
I20260929 22:50:16.771904 15537 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:50:16.762500 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.147.194:59123 (local address 127.14.147.254:35003)
0929 22:50:16.762579 (+ 79us) server_negotiation.cc:206] Beginning negotiation
0929 22:50:16.762581 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 22:50:16.762751 (+ 170us) server_negotiation.cc:407] Connection header received
0929 22:50:16.763713 (+ 962us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 22:50:16.763715 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 22:50:16.763727 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 22:50:16.763778 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0929 22:50:16.764569 (+ 791us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:50:16.765199 (+ 630us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:50:16.766236 (+ 1037us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:50:16.766439 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:50:16.766494 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 22:50:16.768782 (+ 2288us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 22:50:16.768784 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 22:50:16.768784 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 22:50:16.768793 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0929 22:50:16.770363 (+ 1570us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 22:50:16.770836 (+ 473us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 22:50:16.770837 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 22:50:16.770838 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 22:50:16.770901 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 22:50:16.771116 (+ 215us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 22:50:16.771117 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 22:50:16.771118 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 22:50:16.771258 (+ 140us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 22:50:16.771332 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0929 22:50:16.771712 (+ 380us) server_negotiation.cc:299] Negotiation successful
0929 22:50:16.771806 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260929 22:50:16.772004 15825 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:50:16.762437 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.147.254:35003 (local address 127.14.147.194:59123)
0929 22:50:16.762625 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0929 22:50:16.762666 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0929 22:50:16.763603 (+ 937us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 22:50:16.763847 (+ 244us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 22:50:16.763850 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 22:50:16.764140 (+ 290us) client_negotiation.cc:189] Negotiated authn=SASL
0929 22:50:16.764419 (+ 279us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 22:50:16.764427 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 22:50:16.765301 (+ 874us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 22:50:16.765302 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 22:50:16.766145 (+ 843us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 22:50:16.766148 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 22:50:16.766512 (+ 364us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 22:50:16.766514 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 22:50:16.766640 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 22:50:16.767130 (+ 490us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 22:50:16.767138 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0929 22:50:16.768678 (+ 1540us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 22:50:16.770473 (+ 1795us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 22:50:16.770483 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 22:50:16.770485 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 22:50:16.770756 (+ 271us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 22:50:16.770965 (+ 209us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 22:50:16.770965 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 22:50:16.770965 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 22:50:16.771062 (+ 97us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 22:50:16.771350 (+ 288us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 22:50:16.771357 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 22:50:16.771562 (+ 205us) client_negotiation.cc:769] Sending connection context
0929 22:50:16.771654 (+ 92us) client_negotiation.cc:240] Negotiation successful
0929 22:50:16.771814 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":141,"thread_start_us":92,"threads_started":1}
WARNING: no policy specified for kudu/127.14.147.195@KRBTEST.COM; defaulting to no policy
I20260929 22:50:16.772480 15823 heartbeater.cc:347] Connected to a master server at 127.14.147.254:35003
I20260929 22:50:16.772583 15823 heartbeater.cc:464] Registering TS with master...
I20260929 22:50:16.772773 15823 heartbeater.cc:511] Master 127.14.147.254:35003 requested a full tablet report, sending...
I20260929 22:50:16.773124 15498 ts_manager.cc:194] Registered new tserver with Master: fa42e58470984c3c90abc119f317b887 (127.14.147.194:43245)
I20260929 22:50:16.773517 15498 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.147.194@KRBTEST.COM'} at 127.14.147.194:59123
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.147.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.147.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790722213368738-14927-0/krb5kdc/kudu_127.14.147.195.keytab.
Entry for principal kudu/127.14.147.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790722213368738-14927-0/krb5kdc/kudu_127.14.147.195.keytab.
I20260929 22:50:16.786783 14927 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhWOCLk/build/release/bin/kudu
/tmp/dist-test-taskhWOCLk/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790722213368738-14927-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790722213368738-14927-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790722213368738-14927-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790722213368738-14927-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.147.195:0
--local_ip_for_outbound_sockets=127.14.147.195
--webserver_interface=127.14.147.195
--webserver_port=0
--tserver_master_addrs=127.14.147.254:35003
--keytab_file=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790722213368738-14927-0/krb5kdc/kudu_127.14.147.195.keytab
--principal=kudu/127.14.147.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.147.212:34039
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790722213368738-14927-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790722213368738-14927-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790722213368738-14927-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 22:50:16.867580 15828 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 22:50:16.867799 15828 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 22:50:16.867825 15828 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 22:50:16.868978 15828 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 22:50:16.869011 15828 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 22:50:16.869037 15828 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.147.195
I20260929 22:50:16.870520 15828 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.147.212:34039
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790722213368738-14927-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790722213368738-14927-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790722213368738-14927-0/krb5kdc/kudu_127.14.147.195.keytab
--principal=kudu/127.14.147.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.147.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790722213368738-14927-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.14.147.195
--webserver_port=0
--tserver_master_addrs=127.14.147.254:35003
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15828
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.147.195
--log_dir=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790722213368738-14927-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 21:58:54 UTC on ccycloud.quasar-xguoti.root.comops.site
build id 4681390
I20260929 22:50:16.870786 15828 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 22:50:16.870985 15828 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 22:50:16.875672 15834 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 22:50:16.875684 15836 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 22:50:16.875696 15833 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 22:50:16.875914 15828 server_base.cc:1034] running on GCE node
I20260929 22:50:16.876120 15828 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 22:50:16.876358 15828 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 22:50:16.877493 15828 hybrid_clock.cc:648] HybridClock initialized: now 1790722216877481 us; error 29 us; skew 500 ppm
Sep 29 22:50:16 dist-test-slave-98mk krb5kdc[15453](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790722216, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.147.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 22:50:16.880561 15828 init.cc:377] Logged in from keytab as kudu/127.14.147.195@KRBTEST.COM (short username kudu)
I20260929 22:50:16.881085 15828 webserver.cc:460] Webserver started at http://127.14.147.195:42647/ using document root <none> and password file <none>
I20260929 22:50:16.881282 15828 fs_manager.cc:359] Metadata directory not provided
I20260929 22:50:16.881320 15828 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 22:50:16.881412 15828 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 22:50:16.882377 15828 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790722213368738-14927-0/minicluster-data/ts-2/data/instance:
uuid: "02e633eaedcb4e97bd9d8109c53d3b2b"
format_stamp: "Formatted at 2026-09-29 22:50:16 on dist-test-slave-98mk"
server_key: "a11f64fba05b168a882a605656179bc6"
server_key_iv: "023840c46d80d17f5d258b17a59b1045"
server_key_version: "encryptionkey@0"
I20260929 22:50:16.882666 15828 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790722213368738-14927-0/minicluster-data/ts-2/wal/instance:
uuid: "02e633eaedcb4e97bd9d8109c53d3b2b"
format_stamp: "Formatted at 2026-09-29 22:50:16 on dist-test-slave-98mk"
server_key: "a11f64fba05b168a882a605656179bc6"
server_key_iv: "023840c46d80d17f5d258b17a59b1045"
server_key_version: "encryptionkey@0"
I20260929 22:50:16.883890 15828 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260929 22:50:16.884506 15843 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 22:50:16.884661 15828 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260929 22:50:16.884737 15828 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790722213368738-14927-0/minicluster-data/ts-2/data,/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790722213368738-14927-0/minicluster-data/ts-2/wal
uuid: "02e633eaedcb4e97bd9d8109c53d3b2b"
format_stamp: "Formatted at 2026-09-29 22:50:16 on dist-test-slave-98mk"
server_key: "a11f64fba05b168a882a605656179bc6"
server_key_iv: "023840c46d80d17f5d258b17a59b1045"
server_key_version: "encryptionkey@0"
I20260929 22:50:16.884793 15828 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790722213368738-14927-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790722213368738-14927-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790722213368738-14927-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 22:50:16.897787 15828 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 22:50:16.900192 15828 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 22:50:16.900311 15828 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 22:50:16.900522 15828 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 22:50:16.900782 15828 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 22:50:16.900810 15828 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 22:50:16.900832 15828 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 22:50:16.900843 15828 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 22:50:16.906271 15828 rpc_server.cc:307] RPC server started. Bound to: 127.14.147.195:45879
I20260929 22:50:16.906344 15956 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.147.195:45879 every 8 connection(s)
I20260929 22:50:16.906636 15828 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790722213368738-14927-0/minicluster-data/ts-2/data/info.pb
I20260929 22:50:16.911388 14927 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhWOCLk/build/release/bin/kudu as pid 15828
I20260929 22:50:16.911502 14927 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790722213368738-14927-0/minicluster-data/ts-2/wal/instance
I20260929 22:50:16.911665 14927 external_mini_cluster.cc:1428] Setting key 8b354ed18a713ca0a2004a7c7c3db1ec
Sep 29 22:50:16 dist-test-slave-98mk krb5kdc[15453](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790722216, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.147.195@KRBTEST.COM for kudu/127.14.147.254@KRBTEST.COM
I20260929 22:50:16.916055 15537 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:50:16.907321 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.147.195:51287 (local address 127.14.147.254:35003)
0929 22:50:16.907396 (+ 75us) server_negotiation.cc:206] Beginning negotiation
0929 22:50:16.907399 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 22:50:16.907637 (+ 238us) server_negotiation.cc:407] Connection header received
0929 22:50:16.908550 (+ 913us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 22:50:16.908551 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 22:50:16.908562 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 22:50:16.908620 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0929 22:50:16.909351 (+ 731us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:50:16.909876 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:50:16.911016 (+ 1140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:50:16.911250 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:50:16.911317 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 22:50:16.913503 (+ 2186us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 22:50:16.913504 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 22:50:16.913505 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 22:50:16.913512 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0929 22:50:16.914756 (+ 1244us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 22:50:16.915151 (+ 395us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 22:50:16.915152 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 22:50:16.915152 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 22:50:16.915209 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 22:50:16.915448 (+ 239us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 22:50:16.915449 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 22:50:16.915449 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 22:50:16.915565 (+ 116us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 22:50:16.915622 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0929 22:50:16.915880 (+ 258us) server_negotiation.cc:299] Negotiation successful
0929 22:50:16.915959 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260929 22:50:16.916219 15959 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:50:16.907297 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.147.254:35003 (local address 127.14.147.195:51287)
0929 22:50:16.907509 (+ 212us) negotiation.cc:107] Waiting for socket to connect
0929 22:50:16.907551 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0929 22:50:16.908440 (+ 889us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 22:50:16.908688 (+ 248us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 22:50:16.908691 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 22:50:16.909023 (+ 332us) client_negotiation.cc:189] Negotiated authn=SASL
0929 22:50:16.909272 (+ 249us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 22:50:16.909277 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 22:50:16.910026 (+ 749us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 22:50:16.910027 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 22:50:16.910936 (+ 909us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 22:50:16.910939 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 22:50:16.911349 (+ 410us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 22:50:16.911350 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 22:50:16.911517 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 22:50:16.912065 (+ 548us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 22:50:16.912074 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0929 22:50:16.913420 (+ 1346us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 22:50:16.914846 (+ 1426us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 22:50:16.914854 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 22:50:16.914856 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 22:50:16.915081 (+ 225us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 22:50:16.915263 (+ 182us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 22:50:16.915264 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 22:50:16.915265 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 22:50:16.915370 (+ 105us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 22:50:16.915620 (+ 250us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 22:50:16.915627 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 22:50:16.915762 (+ 135us) client_negotiation.cc:769] Sending connection context
0929 22:50:16.915851 (+ 89us) client_negotiation.cc:240] Negotiation successful
0929 22:50:16.916009 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":168,"thread_start_us":79,"threads_started":1}
I20260929 22:50:16.916616 15957 heartbeater.cc:347] Connected to a master server at 127.14.147.254:35003
I20260929 22:50:16.916713 15957 heartbeater.cc:464] Registering TS with master...
I20260929 22:50:16.916885 15957 heartbeater.cc:511] Master 127.14.147.254:35003 requested a full tablet report, sending...
I20260929 22:50:16.917207 15498 ts_manager.cc:194] Registered new tserver with Master: 02e633eaedcb4e97bd9d8109c53d3b2b (127.14.147.195:45879)
I20260929 22:50:16.917543 15498 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.147.195@KRBTEST.COM'} at 127.14.147.195:51287
I20260929 22:50:16.923266 14927 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 22:50:16.930199 15537 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:50:16.924777 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49378 (local address 127.14.147.254:35003)
0929 22:50:16.924835 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0929 22:50:16.924837 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 22:50:16.924894 (+ 57us) server_negotiation.cc:407] Connection header received
0929 22:50:16.924936 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 22:50:16.924937 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 22:50:16.924946 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 22:50:16.924995 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0929 22:50:16.925482 (+ 487us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:50:16.925961 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:50:16.926627 (+ 666us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:50:16.926809 (+ 182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:50:16.926853 (+ 44us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 22:50:16.927903 (+ 1050us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 22:50:16.927904 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 22:50:16.927905 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 22:50:16.927911 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0929 22:50:16.929124 (+ 1213us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 22:50:16.929485 (+ 361us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 22:50:16.929486 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 22:50:16.929486 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 22:50:16.929543 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 22:50:16.929686 (+ 143us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 22:50:16.929686 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 22:50:16.929687 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 22:50:16.929844 (+ 157us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 22:50:16.929918 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0929 22:50:16.930071 (+ 153us) server_negotiation.cc:299] Negotiation successful
0929 22:50:16.930134 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260929 22:50:16.932263 15498 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49378:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 22:50:16.933792 15498 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 22:50:16.939867 15967 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:50:16.935747 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53738 (local address 127.14.147.195:45879)
0929 22:50:16.935935 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0929 22:50:16.935941 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 22:50:16.936075 (+ 134us) server_negotiation.cc:407] Connection header received
0929 22:50:16.936199 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 22:50:16.936202 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 22:50:16.936220 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 22:50:16.936272 (+ 52us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 22:50:16.936724 (+ 452us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:50:16.937760 (+ 1036us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:50:16.938799 (+ 1039us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:50:16.939397 (+ 598us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:50:16.939497 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 22:50:16.939555 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0929 22:50:16.939726 (+ 171us) server_negotiation.cc:299] Negotiation successful
0929 22:50:16.939775 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":144,"thread_start_us":96,"threads_started":1}
I20260929 22:50:16.940088 15968 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:50:16.935663 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.147.193:39605 (local address 127.0.0.1:43456)
0929 22:50:16.935971 (+ 308us) negotiation.cc:107] Waiting for socket to connect
0929 22:50:16.936013 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0929 22:50:16.936088 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 22:50:16.936381 (+ 293us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 22:50:16.936384 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 22:50:16.936386 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 22:50:16.936448 (+ 62us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 22:50:16.936450 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 22:50:16.937723 (+ 1273us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 22:50:16.937723 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 22:50:16.939045 (+ 1322us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 22:50:16.939048 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 22:50:16.939853 (+ 805us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 22:50:16.939854 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 22:50:16.939909 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 22:50:16.939913 (+ 4us) client_negotiation.cc:769] Sending connection context
0929 22:50:16.939941 (+ 28us) client_negotiation.cc:240] Negotiation successful
0929 22:50:16.940010 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":273,"mutex_wait_us":47,"thread_start_us":80,"threads_started":1}
I20260929 22:50:16.940269 15966 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:50:16.935665 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.147.195:45879 (local address 127.0.0.1:53738)
0929 22:50:16.935989 (+ 324us) negotiation.cc:107] Waiting for socket to connect
0929 22:50:16.936013 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0929 22:50:16.936088 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 22:50:16.936509 (+ 421us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 22:50:16.936510 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 22:50:16.936512 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 22:50:16.936567 (+ 55us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 22:50:16.936568 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 22:50:16.937918 (+ 1350us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 22:50:16.937919 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 22:50:16.938714 (+ 795us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 22:50:16.938717 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 22:50:16.939503 (+ 786us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 22:50:16.939504 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 22:50:16.939589 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 22:50:16.939603 (+ 14us) client_negotiation.cc:769] Sending connection context
0929 22:50:16.940051 (+ 448us) client_negotiation.cc:240] Negotiation successful
0929 22:50:16.940088 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":314,"mutex_wait_us":2,"thread_start_us":80,"threads_started":1}
I20260929 22:50:16.940290 15965 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:50:16.935866 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.147.194:43245 (local address 127.0.0.1:51804)
0929 22:50:16.936379 (+ 513us) negotiation.cc:107] Waiting for socket to connect
0929 22:50:16.936388 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0929 22:50:16.936442 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 22:50:16.936693 (+ 251us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 22:50:16.936694 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 22:50:16.936696 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 22:50:16.936732 (+ 36us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 22:50:16.936734 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 22:50:16.937658 (+ 924us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 22:50:16.937659 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 22:50:16.939045 (+ 1386us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 22:50:16.939048 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 22:50:16.940032 (+ 984us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 22:50:16.940033 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 22:50:16.940093 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 22:50:16.940097 (+ 4us) client_negotiation.cc:769] Sending connection context
0929 22:50:16.940140 (+ 43us) client_negotiation.cc:240] Negotiation successful
0929 22:50:16.940187 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":499,"mutex_wait_us":49,"thread_start_us":46,"threads_started":1}
I20260929 22:50:16.940510 15970 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:50:16.936041 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51804 (local address 127.14.147.194:43245)
0929 22:50:16.936200 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0929 22:50:16.936204 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 22:50:16.936453 (+ 249us) server_negotiation.cc:407] Connection header received
0929 22:50:16.936499 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 22:50:16.936502 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 22:50:16.936518 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 22:50:16.936602 (+ 84us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 22:50:16.936786 (+ 184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:50:16.937537 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:50:16.939138 (+ 1601us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:50:16.939886 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:50:16.939947 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 22:50:16.940000 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0929 22:50:16.940385 (+ 385us) server_negotiation.cc:299] Negotiation successful
0929 22:50:16.940437 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":132,"thread_start_us":86,"threads_started":1}
I20260929 22:50:16.940526 15969 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:50:16.935853 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43456 (local address 127.14.147.193:39605)
0929 22:50:16.936230 (+ 377us) server_negotiation.cc:206] Beginning negotiation
0929 22:50:16.936234 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 22:50:16.936245 (+ 11us) server_negotiation.cc:407] Connection header received
0929 22:50:16.936281 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 22:50:16.936283 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 22:50:16.936297 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 22:50:16.936365 (+ 68us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 22:50:16.936532 (+ 167us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:50:16.937636 (+ 1104us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:50:16.939148 (+ 1512us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:50:16.939769 (+ 621us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:50:16.940302 (+ 533us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 22:50:16.940368 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0929 22:50:16.940394 (+ 26us) server_negotiation.cc:299] Negotiation successful
0929 22:50:16.940440 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":346,"thread_start_us":70,"threads_started":1}
I20260929 22:50:16.941190 15891 tablet_service.cc:1467] Processing CreateTablet for tablet a7055914691d47f88ee24bc4d3652bf6 (DEFAULT_TABLE table=test-table [id=06f62daaa2ce4ea085df4ddaa2501c7f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 22:50:16.941179 15757 tablet_service.cc:1467] Processing CreateTablet for tablet a7055914691d47f88ee24bc4d3652bf6 (DEFAULT_TABLE table=test-table [id=06f62daaa2ce4ea085df4ddaa2501c7f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 22:50:16.941222 15623 tablet_service.cc:1467] Processing CreateTablet for tablet a7055914691d47f88ee24bc4d3652bf6 (DEFAULT_TABLE table=test-table [id=06f62daaa2ce4ea085df4ddaa2501c7f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 22:50:16.941515 15891 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a7055914691d47f88ee24bc4d3652bf6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 22:50:16.941515 15623 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a7055914691d47f88ee24bc4d3652bf6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 22:50:16.941519 15757 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a7055914691d47f88ee24bc4d3652bf6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 22:50:16.944176 15971 tablet_bootstrap.cc:492] T a7055914691d47f88ee24bc4d3652bf6 P 02e633eaedcb4e97bd9d8109c53d3b2b: Bootstrap starting.
I20260929 22:50:16.944792 15972 tablet_bootstrap.cc:492] T a7055914691d47f88ee24bc4d3652bf6 P fa42e58470984c3c90abc119f317b887: Bootstrap starting.
I20260929 22:50:16.944845 15971 tablet_bootstrap.cc:654] T a7055914691d47f88ee24bc4d3652bf6 P 02e633eaedcb4e97bd9d8109c53d3b2b: Neither blocks nor log segments found. Creating new log.
I20260929 22:50:16.945497 15971 log.cc:824] T a7055914691d47f88ee24bc4d3652bf6 P 02e633eaedcb4e97bd9d8109c53d3b2b: Log is configured to *not* fsync() on all Append() calls
I20260929 22:50:16.945519 15972 tablet_bootstrap.cc:654] T a7055914691d47f88ee24bc4d3652bf6 P fa42e58470984c3c90abc119f317b887: Neither blocks nor log segments found. Creating new log.
I20260929 22:50:16.945848 15973 tablet_bootstrap.cc:492] T a7055914691d47f88ee24bc4d3652bf6 P 461a4dc5d5fa4b9f8cac432d45aa8913: Bootstrap starting.
I20260929 22:50:16.946117 15972 log.cc:824] T a7055914691d47f88ee24bc4d3652bf6 P fa42e58470984c3c90abc119f317b887: Log is configured to *not* fsync() on all Append() calls
I20260929 22:50:16.946291 15971 tablet_bootstrap.cc:492] T a7055914691d47f88ee24bc4d3652bf6 P 02e633eaedcb4e97bd9d8109c53d3b2b: No bootstrap required, opened a new log
I20260929 22:50:16.946358 15971 ts_tablet_manager.cc:1397] T a7055914691d47f88ee24bc4d3652bf6 P 02e633eaedcb4e97bd9d8109c53d3b2b: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260929 22:50:16.946547 15973 tablet_bootstrap.cc:654] T a7055914691d47f88ee24bc4d3652bf6 P 461a4dc5d5fa4b9f8cac432d45aa8913: Neither blocks nor log segments found. Creating new log.
I20260929 22:50:16.946976 15973 log.cc:824] T a7055914691d47f88ee24bc4d3652bf6 P 461a4dc5d5fa4b9f8cac432d45aa8913: Log is configured to *not* fsync() on all Append() calls
I20260929 22:50:16.947701 15972 tablet_bootstrap.cc:492] T a7055914691d47f88ee24bc4d3652bf6 P fa42e58470984c3c90abc119f317b887: No bootstrap required, opened a new log
I20260929 22:50:16.947717 15973 tablet_bootstrap.cc:492] T a7055914691d47f88ee24bc4d3652bf6 P 461a4dc5d5fa4b9f8cac432d45aa8913: No bootstrap required, opened a new log
I20260929 22:50:16.947760 15972 ts_tablet_manager.cc:1397] T a7055914691d47f88ee24bc4d3652bf6 P fa42e58470984c3c90abc119f317b887: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260929 22:50:16.947773 15973 ts_tablet_manager.cc:1397] T a7055914691d47f88ee24bc4d3652bf6 P 461a4dc5d5fa4b9f8cac432d45aa8913: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260929 22:50:16.948036 15971 raft_consensus.cc:348] T a7055914691d47f88ee24bc4d3652bf6 P 02e633eaedcb4e97bd9d8109c53d3b2b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "461a4dc5d5fa4b9f8cac432d45aa8913" member_type: VOTER last_known_addr { host: "127.14.147.193" port: 39605 } } peers { permanent_uuid: "02e633eaedcb4e97bd9d8109c53d3b2b" member_type: VOTER last_known_addr { host: "127.14.147.195" port: 45879 } } peers { permanent_uuid: "fa42e58470984c3c90abc119f317b887" member_type: VOTER last_known_addr { host: "127.14.147.194" port: 43245 } }
I20260929 22:50:16.948232 15971 raft_consensus.cc:374] T a7055914691d47f88ee24bc4d3652bf6 P 02e633eaedcb4e97bd9d8109c53d3b2b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 22:50:16.948264 15971 raft_consensus.cc:729] T a7055914691d47f88ee24bc4d3652bf6 P 02e633eaedcb4e97bd9d8109c53d3b2b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 02e633eaedcb4e97bd9d8109c53d3b2b, State: Initialized, Role: FOLLOWER
I20260929 22:50:16.948378 15971 consensus_queue.cc:260] T a7055914691d47f88ee24bc4d3652bf6 P 02e633eaedcb4e97bd9d8109c53d3b2b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "461a4dc5d5fa4b9f8cac432d45aa8913" member_type: VOTER last_known_addr { host: "127.14.147.193" port: 39605 } } peers { permanent_uuid: "02e633eaedcb4e97bd9d8109c53d3b2b" member_type: VOTER last_known_addr { host: "127.14.147.195" port: 45879 } } peers { permanent_uuid: "fa42e58470984c3c90abc119f317b887" member_type: VOTER last_known_addr { host: "127.14.147.194" port: 43245 } }
I20260929 22:50:16.948659 15971 ts_tablet_manager.cc:1428] T a7055914691d47f88ee24bc4d3652bf6 P 02e633eaedcb4e97bd9d8109c53d3b2b: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260929 22:50:16.948669 15957 heartbeater.cc:503] Master 127.14.147.254:35003 was elected leader, sending a full tablet report...
I20260929 22:50:16.949391 15972 raft_consensus.cc:348] T a7055914691d47f88ee24bc4d3652bf6 P fa42e58470984c3c90abc119f317b887 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "461a4dc5d5fa4b9f8cac432d45aa8913" member_type: VOTER last_known_addr { host: "127.14.147.193" port: 39605 } } peers { permanent_uuid: "02e633eaedcb4e97bd9d8109c53d3b2b" member_type: VOTER last_known_addr { host: "127.14.147.195" port: 45879 } } peers { permanent_uuid: "fa42e58470984c3c90abc119f317b887" member_type: VOTER last_known_addr { host: "127.14.147.194" port: 43245 } }
I20260929 22:50:16.949528 15972 raft_consensus.cc:374] T a7055914691d47f88ee24bc4d3652bf6 P fa42e58470984c3c90abc119f317b887 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 22:50:16.949560 15972 raft_consensus.cc:729] T a7055914691d47f88ee24bc4d3652bf6 P fa42e58470984c3c90abc119f317b887 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fa42e58470984c3c90abc119f317b887, State: Initialized, Role: FOLLOWER
I20260929 22:50:16.949678 15972 consensus_queue.cc:260] T a7055914691d47f88ee24bc4d3652bf6 P fa42e58470984c3c90abc119f317b887 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "461a4dc5d5fa4b9f8cac432d45aa8913" member_type: VOTER last_known_addr { host: "127.14.147.193" port: 39605 } } peers { permanent_uuid: "02e633eaedcb4e97bd9d8109c53d3b2b" member_type: VOTER last_known_addr { host: "127.14.147.195" port: 45879 } } peers { permanent_uuid: "fa42e58470984c3c90abc119f317b887" member_type: VOTER last_known_addr { host: "127.14.147.194" port: 43245 } }
I20260929 22:50:16.949472 15973 raft_consensus.cc:348] T a7055914691d47f88ee24bc4d3652bf6 P 461a4dc5d5fa4b9f8cac432d45aa8913 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "461a4dc5d5fa4b9f8cac432d45aa8913" member_type: VOTER last_known_addr { host: "127.14.147.193" port: 39605 } } peers { permanent_uuid: "02e633eaedcb4e97bd9d8109c53d3b2b" member_type: VOTER last_known_addr { host: "127.14.147.195" port: 45879 } } peers { permanent_uuid: "fa42e58470984c3c90abc119f317b887" member_type: VOTER last_known_addr { host: "127.14.147.194" port: 43245 } }
I20260929 22:50:16.949805 15973 raft_consensus.cc:374] T a7055914691d47f88ee24bc4d3652bf6 P 461a4dc5d5fa4b9f8cac432d45aa8913 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 22:50:16.949873 15973 raft_consensus.cc:729] T a7055914691d47f88ee24bc4d3652bf6 P 461a4dc5d5fa4b9f8cac432d45aa8913 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 461a4dc5d5fa4b9f8cac432d45aa8913, State: Initialized, Role: FOLLOWER
I20260929 22:50:16.949952 15972 ts_tablet_manager.cc:1428] T a7055914691d47f88ee24bc4d3652bf6 P fa42e58470984c3c90abc119f317b887: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260929 22:50:16.949997 15823 heartbeater.cc:503] Master 127.14.147.254:35003 was elected leader, sending a full tablet report...
I20260929 22:50:16.950008 15973 consensus_queue.cc:260] T a7055914691d47f88ee24bc4d3652bf6 P 461a4dc5d5fa4b9f8cac432d45aa8913 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "461a4dc5d5fa4b9f8cac432d45aa8913" member_type: VOTER last_known_addr { host: "127.14.147.193" port: 39605 } } peers { permanent_uuid: "02e633eaedcb4e97bd9d8109c53d3b2b" member_type: VOTER last_known_addr { host: "127.14.147.195" port: 45879 } } peers { permanent_uuid: "fa42e58470984c3c90abc119f317b887" member_type: VOTER last_known_addr { host: "127.14.147.194" port: 43245 } }
I20260929 22:50:16.950261 15973 ts_tablet_manager.cc:1428] T a7055914691d47f88ee24bc4d3652bf6 P 461a4dc5d5fa4b9f8cac432d45aa8913: Time spent starting tablet: real 0.002s user 0.000s sys 0.003s
I20260929 22:50:16.950376 15689 heartbeater.cc:503] Master 127.14.147.254:35003 was elected leader, sending a full tablet report...
W20260929 22:50:17.012389 15824 tablet.cc:2369] T a7055914691d47f88ee24bc4d3652bf6 P fa42e58470984c3c90abc119f317b887: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 22:50:17.062965 15977 raft_consensus.cc:482] T a7055914691d47f88ee24bc4d3652bf6 P 02e633eaedcb4e97bd9d8109c53d3b2b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 22:50:17.063103 15977 raft_consensus.cc:504] T a7055914691d47f88ee24bc4d3652bf6 P 02e633eaedcb4e97bd9d8109c53d3b2b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "461a4dc5d5fa4b9f8cac432d45aa8913" member_type: VOTER last_known_addr { host: "127.14.147.193" port: 39605 } } peers { permanent_uuid: "02e633eaedcb4e97bd9d8109c53d3b2b" member_type: VOTER last_known_addr { host: "127.14.147.195" port: 45879 } } peers { permanent_uuid: "fa42e58470984c3c90abc119f317b887" member_type: VOTER last_known_addr { host: "127.14.147.194" port: 43245 } }
I20260929 22:50:17.063505 15977 leader_election.cc:288] T a7055914691d47f88ee24bc4d3652bf6 P 02e633eaedcb4e97bd9d8109c53d3b2b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 461a4dc5d5fa4b9f8cac432d45aa8913 (127.14.147.193:39605), fa42e58470984c3c90abc119f317b887 (127.14.147.194:43245)
I20260929 22:50:17.066468 15959 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:50:17.063730 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.147.193:39605 (local address 127.14.147.195:50075)
0929 22:50:17.063792 (+ 62us) negotiation.cc:107] Waiting for socket to connect
0929 22:50:17.063803 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0929 22:50:17.063848 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 22:50:17.064010 (+ 162us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 22:50:17.064011 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 22:50:17.064013 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 22:50:17.064095 (+ 82us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 22:50:17.064097 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 22:50:17.064773 (+ 676us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 22:50:17.064774 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 22:50:17.065583 (+ 809us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 22:50:17.065586 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 22:50:17.066194 (+ 608us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 22:50:17.066195 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 22:50:17.066291 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 22:50:17.066296 (+ 5us) client_negotiation.cc:769] Sending connection context
0929 22:50:17.066342 (+ 46us) client_negotiation.cc:240] Negotiation successful
0929 22:50:17.066388 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":44}
I20260929 22:50:17.066537 15969 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:50:17.063798 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.147.195:50075 (local address 127.14.147.193:39605)
0929 22:50:17.063856 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0929 22:50:17.063858 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 22:50:17.063868 (+ 10us) server_negotiation.cc:407] Connection header received
0929 22:50:17.063900 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 22:50:17.063901 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 22:50:17.063912 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 22:50:17.063994 (+ 82us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 22:50:17.064160 (+ 166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:50:17.064675 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:50:17.065671 (+ 996us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:50:17.066093 (+ 422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:50:17.066185 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 22:50:17.066196 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0929 22:50:17.066406 (+ 210us) server_negotiation.cc:299] Negotiation successful
0929 22:50:17.066459 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260929 22:50:17.066720 15970 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:50:17.063797 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.147.195:47173 (local address 127.14.147.194:43245)
0929 22:50:17.063847 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0929 22:50:17.063849 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 22:50:17.064024 (+ 175us) server_negotiation.cc:407] Connection header received
0929 22:50:17.064061 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 22:50:17.064062 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 22:50:17.064073 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 22:50:17.064129 (+ 56us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 22:50:17.064309 (+ 180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:50:17.064816 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:50:17.065767 (+ 951us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:50:17.066244 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:50:17.066322 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 22:50:17.066332 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0929 22:50:17.066539 (+ 207us) server_negotiation.cc:299] Negotiation successful
0929 22:50:17.066630 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260929 22:50:17.067021 15980 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:50:17.063730 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.147.194:43245 (local address 127.14.147.195:47173)
0929 22:50:17.063947 (+ 217us) negotiation.cc:107] Waiting for socket to connect
0929 22:50:17.063957 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0929 22:50:17.064015 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 22:50:17.064139 (+ 124us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 22:50:17.064140 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 22:50:17.064141 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 22:50:17.064214 (+ 73us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 22:50:17.064215 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 22:50:17.064904 (+ 689us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 22:50:17.064904 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 22:50:17.065687 (+ 783us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 22:50:17.065690 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 22:50:17.066316 (+ 626us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 22:50:17.066317 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 22:50:17.066410 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 22:50:17.066413 (+ 3us) client_negotiation.cc:769] Sending connection context
0929 22:50:17.066880 (+ 467us) client_negotiation.cc:240] Negotiation successful
0929 22:50:17.066925 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":198,"mutex_wait_us":23,"thread_start_us":134,"threads_started":1}
I20260929 22:50:17.067016 15643 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a7055914691d47f88ee24bc4d3652bf6" candidate_uuid: "02e633eaedcb4e97bd9d8109c53d3b2b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "461a4dc5d5fa4b9f8cac432d45aa8913" is_pre_election: true
I20260929 22:50:17.067183 15643 raft_consensus.cc:2454] T a7055914691d47f88ee24bc4d3652bf6 P 461a4dc5d5fa4b9f8cac432d45aa8913 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 02e633eaedcb4e97bd9d8109c53d3b2b in term 0.
I20260929 22:50:17.067457 15846 leader_election.cc:302] T a7055914691d47f88ee24bc4d3652bf6 P 02e633eaedcb4e97bd9d8109c53d3b2b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 02e633eaedcb4e97bd9d8109c53d3b2b, 461a4dc5d5fa4b9f8cac432d45aa8913; no voters:
I20260929 22:50:17.067399 15777 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a7055914691d47f88ee24bc4d3652bf6" candidate_uuid: "02e633eaedcb4e97bd9d8109c53d3b2b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fa42e58470984c3c90abc119f317b887" is_pre_election: true
I20260929 22:50:17.067584 15777 raft_consensus.cc:2454] T a7055914691d47f88ee24bc4d3652bf6 P fa42e58470984c3c90abc119f317b887 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 02e633eaedcb4e97bd9d8109c53d3b2b in term 0.
I20260929 22:50:17.067593 15977 raft_consensus.cc:2781] T a7055914691d47f88ee24bc4d3652bf6 P 02e633eaedcb4e97bd9d8109c53d3b2b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 22:50:17.067670 15977 raft_consensus.cc:482] T a7055914691d47f88ee24bc4d3652bf6 P 02e633eaedcb4e97bd9d8109c53d3b2b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 22:50:17.067692 15977 raft_consensus.cc:3037] T a7055914691d47f88ee24bc4d3652bf6 P 02e633eaedcb4e97bd9d8109c53d3b2b [term 0 FOLLOWER]: Advancing to term 1
I20260929 22:50:17.068392 15977 raft_consensus.cc:504] T a7055914691d47f88ee24bc4d3652bf6 P 02e633eaedcb4e97bd9d8109c53d3b2b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "461a4dc5d5fa4b9f8cac432d45aa8913" member_type: VOTER last_known_addr { host: "127.14.147.193" port: 39605 } } peers { permanent_uuid: "02e633eaedcb4e97bd9d8109c53d3b2b" member_type: VOTER last_known_addr { host: "127.14.147.195" port: 45879 } } peers { permanent_uuid: "fa42e58470984c3c90abc119f317b887" member_type: VOTER last_known_addr { host: "127.14.147.194" port: 43245 } }
I20260929 22:50:17.068519 15977 leader_election.cc:288] T a7055914691d47f88ee24bc4d3652bf6 P 02e633eaedcb4e97bd9d8109c53d3b2b [CANDIDATE]: Term 1 election: Requested vote from peers 461a4dc5d5fa4b9f8cac432d45aa8913 (127.14.147.193:39605), fa42e58470984c3c90abc119f317b887 (127.14.147.194:43245)
I20260929 22:50:17.068682 15777 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a7055914691d47f88ee24bc4d3652bf6" candidate_uuid: "02e633eaedcb4e97bd9d8109c53d3b2b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fa42e58470984c3c90abc119f317b887"
I20260929 22:50:17.068696 15643 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a7055914691d47f88ee24bc4d3652bf6" candidate_uuid: "02e633eaedcb4e97bd9d8109c53d3b2b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "461a4dc5d5fa4b9f8cac432d45aa8913"
I20260929 22:50:17.068765 15643 raft_consensus.cc:3037] T a7055914691d47f88ee24bc4d3652bf6 P 461a4dc5d5fa4b9f8cac432d45aa8913 [term 0 FOLLOWER]: Advancing to term 1
I20260929 22:50:17.068768 15777 raft_consensus.cc:3037] T a7055914691d47f88ee24bc4d3652bf6 P fa42e58470984c3c90abc119f317b887 [term 0 FOLLOWER]: Advancing to term 1
I20260929 22:50:17.069612 15777 raft_consensus.cc:2454] T a7055914691d47f88ee24bc4d3652bf6 P fa42e58470984c3c90abc119f317b887 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 02e633eaedcb4e97bd9d8109c53d3b2b in term 1.
I20260929 22:50:17.069612 15643 raft_consensus.cc:2454] T a7055914691d47f88ee24bc4d3652bf6 P 461a4dc5d5fa4b9f8cac432d45aa8913 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 02e633eaedcb4e97bd9d8109c53d3b2b in term 1.
I20260929 22:50:17.069777 15846 leader_election.cc:302] T a7055914691d47f88ee24bc4d3652bf6 P 02e633eaedcb4e97bd9d8109c53d3b2b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 02e633eaedcb4e97bd9d8109c53d3b2b, 461a4dc5d5fa4b9f8cac432d45aa8913; no voters:
I20260929 22:50:17.069891 15977 raft_consensus.cc:2781] T a7055914691d47f88ee24bc4d3652bf6 P 02e633eaedcb4e97bd9d8109c53d3b2b [term 1 FOLLOWER]: Leader election won for term 1
I20260929 22:50:17.070063 15977 raft_consensus.cc:686] T a7055914691d47f88ee24bc4d3652bf6 P 02e633eaedcb4e97bd9d8109c53d3b2b [term 1 LEADER]: Becoming Leader. State: Replica: 02e633eaedcb4e97bd9d8109c53d3b2b, State: Running, Role: LEADER
I20260929 22:50:17.070171 15977 consensus_queue.cc:237] T a7055914691d47f88ee24bc4d3652bf6 P 02e633eaedcb4e97bd9d8109c53d3b2b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "461a4dc5d5fa4b9f8cac432d45aa8913" member_type: VOTER last_known_addr { host: "127.14.147.193" port: 39605 } } peers { permanent_uuid: "02e633eaedcb4e97bd9d8109c53d3b2b" member_type: VOTER last_known_addr { host: "127.14.147.195" port: 45879 } } peers { permanent_uuid: "fa42e58470984c3c90abc119f317b887" member_type: VOTER last_known_addr { host: "127.14.147.194" port: 43245 } }
I20260929 22:50:17.071076 15496 catalog_manager.cc:5495] T a7055914691d47f88ee24bc4d3652bf6 P 02e633eaedcb4e97bd9d8109c53d3b2b reported cstate change: term changed from 0 to 1, leader changed from <none> to 02e633eaedcb4e97bd9d8109c53d3b2b (127.14.147.195). New cstate: current_term: 1 leader_uuid: "02e633eaedcb4e97bd9d8109c53d3b2b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "461a4dc5d5fa4b9f8cac432d45aa8913" member_type: VOTER last_known_addr { host: "127.14.147.193" port: 39605 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "02e633eaedcb4e97bd9d8109c53d3b2b" member_type: VOTER last_known_addr { host: "127.14.147.195" port: 45879 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fa42e58470984c3c90abc119f317b887" member_type: VOTER last_known_addr { host: "127.14.147.194" port: 43245 } health_report { overall_health: UNKNOWN } } }
Sep 29 22:50:17 dist-test-slave-98mk krb5kdc[15453](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790722216, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.147.193@KRBTEST.COM
I20260929 22:50:17.093737 15969 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:50:17.085945 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43472 (local address 127.14.147.193:39605)
0929 22:50:17.086007 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0929 22:50:17.086008 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0929 22:50:17.086022 (+ 14us) server_negotiation.cc:407] Connection header received
0929 22:50:17.086103 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 22:50:17.086104 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 22:50:17.086114 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 22:50:17.086161 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0929 22:50:17.086675 (+ 514us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:50:17.087151 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:50:17.087968 (+ 817us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:50:17.088168 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:50:17.088240 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 22:50:17.090591 (+ 2351us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 22:50:17.090605 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 22:50:17.090606 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 22:50:17.090614 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0929 22:50:17.092596 (+ 1982us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 22:50:17.092992 (+ 396us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 22:50:17.092995 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 22:50:17.092995 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 22:50:17.093057 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 22:50:17.093236 (+ 179us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 22:50:17.093237 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 22:50:17.093237 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 22:50:17.093399 (+ 162us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 22:50:17.093520 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0929 22:50:17.093573 (+ 53us) server_negotiation.cc:299] Negotiation successful
0929 22:50:17.093645 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
Sep 29 22:50:17 dist-test-slave-98mk krb5kdc[15453](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790722217, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
W20260929 22:50:17.106191 15690 tablet.cc:2369] T a7055914691d47f88ee24bc4d3652bf6 P 461a4dc5d5fa4b9f8cac432d45aa8913: Can't schedule compaction. Clean time has not been advanced past its initial value.
Sep 29 22:50:17 dist-test-slave-98mk krb5kdc[15453](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790722217, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.14.147.193@KRBTEST.COM
I20260929 22:50:17.116392 15969 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:50:17.109368 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43478 (local address 127.14.147.193:39605)
0929 22:50:17.109443 (+ 75us) server_negotiation.cc:206] Beginning negotiation
0929 22:50:17.109445 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 22:50:17.109501 (+ 56us) server_negotiation.cc:407] Connection header received
0929 22:50:17.109554 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 22:50:17.109555 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 22:50:17.109565 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 22:50:17.109612 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0929 22:50:17.110201 (+ 589us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:50:17.110679 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:50:17.111372 (+ 693us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:50:17.111627 (+ 255us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:50:17.111677 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 22:50:17.114148 (+ 2471us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 22:50:17.114150 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 22:50:17.114150 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 22:50:17.114158 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0929 22:50:17.115277 (+ 1119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 22:50:17.115628 (+ 351us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 22:50:17.115629 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 22:50:17.115630 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 22:50:17.115682 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 22:50:17.115896 (+ 214us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 22:50:17.115897 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 22:50:17.115897 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 22:50:17.116014 (+ 117us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 22:50:17.116111 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0929 22:50:17.116253 (+ 142us) server_negotiation.cc:299] Negotiation successful
0929 22:50:17.116321 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
Sep 29 22:50:17 dist-test-slave-98mk krb5kdc[15453](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790722217, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 29 22:50:17 dist-test-slave-98mk krb5kdc[15453](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790722217, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.147.193@KRBTEST.COM
I20260929 22:50:17.137207 15969 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:50:17.131013 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43488 (local address 127.14.147.193:39605)
0929 22:50:17.131073 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0929 22:50:17.131076 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 22:50:17.131163 (+ 87us) server_negotiation.cc:407] Connection header received
0929 22:50:17.131214 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 22:50:17.131215 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 22:50:17.131225 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 22:50:17.131272 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0929 22:50:17.131853 (+ 581us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:50:17.132321 (+ 468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:50:17.133103 (+ 782us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:50:17.133286 (+ 183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:50:17.133328 (+ 42us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 22:50:17.135327 (+ 1999us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 22:50:17.135328 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 22:50:17.135329 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 22:50:17.135336 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0929 22:50:17.136420 (+ 1084us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 22:50:17.136695 (+ 275us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 22:50:17.136695 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 22:50:17.136696 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 22:50:17.136739 (+ 43us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 22:50:17.136866 (+ 127us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 22:50:17.136866 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 22:50:17.136866 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 22:50:17.136947 (+ 81us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 22:50:17.137009 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0929 22:50:17.137085 (+ 76us) server_negotiation.cc:299] Negotiation successful
0929 22:50:17.137139 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
W20260929 22:50:17.137634 15603 tablet_service.cc:2908] Rejecting scan request for tablet a7055914691d47f88ee24bc4d3652bf6: Uninitialized: clean time has not yet been initialized
I20260929 22:50:17.138368 14927 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhWOCLk/build/release/bin/kudu with pid 15560
I20260929 22:50:17.142922 14927 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhWOCLk/build/release/bin/kudu with pid 15694
I20260929 22:50:17.147086 14927 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhWOCLk/build/release/bin/kudu with pid 15828
I20260929 22:50:17.151187 14927 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhWOCLk/build/release/bin/kudu with pid 15466
2026-09-29T22:50:17Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnChecksum (1948 ms)
[ RUN ] SecurityITest.SmokeTestAsAuthorizedUser
Loading random data
Initializing database '/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790722213368738-14927-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 22:50:17 dist-test-slave-98mk krb5kdc[15995](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 22:50:17 dist-test-slave-98mk krb5kdc[15995](info): set up 2 sockets
Sep 29 22:50:17 dist-test-slave-98mk krb5kdc[15995](info): commencing operation
krb5kdc: starting...
W20260929 22:50:18.111694 14927 mini_kdc.cc:121] Time spent starting KDC: real 0.941s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790722213368738-14927-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790722213368738-14927-0/krb5kdc/test-user.keytab.
Sep 29 22:50:18 dist-test-slave-98mk krb5kdc[15995](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790722218, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T22:50:18Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T22:50:18Z Disabled control of system clock
WARNING: no policy specified for kudu/127.14.147.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.147.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.147.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790722213368738-14927-0/krb5kdc/kudu_127.14.147.254.keytab.
Entry for principal kudu/127.14.147.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790722213368738-14927-0/krb5kdc/kudu_127.14.147.254.keytab.
I20260929 22:50:18.225606 14927 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhWOCLk/build/release/bin/kudu
/tmp/dist-test-taskhWOCLk/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790722213368738-14927-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790722213368738-14927-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790722213368738-14927-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790722213368738-14927-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.147.254:46225
--webserver_interface=127.14.147.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790722213368738-14927-0/krb5kdc/kudu_127.14.147.254.keytab
--principal=kudu/127.14.147.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.147.212:36535
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.147.254:46225
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790722213368738-14927-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790722213368738-14927-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790722213368738-14927-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 22:50:18.307448 16008 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 22:50:18.307691 16008 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 22:50:18.307725 16008 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 22:50:18.308944 16008 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 22:50:18.308997 16008 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260929 22:50:18.309013 16008 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 22:50:18.309031 16008 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 22:50:18.309046 16008 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 22:50:18.309062 16008 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 22:50:18.310650 16008 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.147.212:36535
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790722213368738-14927-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790722213368738-14927-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.14.147.254:46225
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790722213368738-14927-0/krb5kdc/kudu_127.14.147.254.keytab
--principal=kudu/127.14.147.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.147.254:46225
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790722213368738-14927-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.14.147.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16008
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790722213368738-14927-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 21:58:54 UTC on ccycloud.quasar-xguoti.root.comops.site
build id 4681390
I20260929 22:50:18.310925 16008 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 22:50:18.311144 16008 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260929 22:50:18.315742 16008 server_base.cc:1034] running on GCE node
W20260929 22:50:18.315693 16013 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 22:50:18.315677 16016 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 22:50:18.315680 16014 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 22:50:18.316164 16008 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 22:50:18.316435 16008 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 22:50:18.317598 16008 hybrid_clock.cc:648] HybridClock initialized: now 1790722218317583 us; error 29 us; skew 500 ppm
Sep 29 22:50:18 dist-test-slave-98mk krb5kdc[15995](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790722218, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.147.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 22:50:18.320098 16008 init.cc:377] Logged in from keytab as kudu/127.14.147.254@KRBTEST.COM (short username kudu)
I20260929 22:50:18.320526 16008 webserver.cc:460] Webserver started at http://127.14.147.254:34747/ using document root <none> and password file <none>
I20260929 22:50:18.320731 16008 fs_manager.cc:359] Metadata directory not provided
I20260929 22:50:18.320777 16008 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 22:50:18.320899 16008 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 22:50:18.321936 16008 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790722213368738-14927-0/minicluster-data/master-0/data/instance:
uuid: "d20dfacee9bd43d3b147457b590bc600"
format_stamp: "Formatted at 2026-09-29 22:50:18 on dist-test-slave-98mk"
server_key: "f626a30bcb5bb571e9daa00f165019b3"
server_key_iv: "88f96f377aefe4150b50e996027b1b42"
server_key_version: "encryptionkey@0"
I20260929 22:50:18.322268 16008 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790722213368738-14927-0/minicluster-data/master-0/wal/instance:
uuid: "d20dfacee9bd43d3b147457b590bc600"
format_stamp: "Formatted at 2026-09-29 22:50:18 on dist-test-slave-98mk"
server_key: "f626a30bcb5bb571e9daa00f165019b3"
server_key_iv: "88f96f377aefe4150b50e996027b1b42"
server_key_version: "encryptionkey@0"
I20260929 22:50:18.323552 16008 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260929 22:50:18.324329 16023 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 22:50:18.324488 16008 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260929 22:50:18.324571 16008 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790722213368738-14927-0/minicluster-data/master-0/data,/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790722213368738-14927-0/minicluster-data/master-0/wal
uuid: "d20dfacee9bd43d3b147457b590bc600"
format_stamp: "Formatted at 2026-09-29 22:50:18 on dist-test-slave-98mk"
server_key: "f626a30bcb5bb571e9daa00f165019b3"
server_key_iv: "88f96f377aefe4150b50e996027b1b42"
server_key_version: "encryptionkey@0"
I20260929 22:50:18.324642 16008 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790722213368738-14927-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790722213368738-14927-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790722213368738-14927-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 22:50:18.345435 16008 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 22:50:18.347697 16008 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 22:50:18.347846 16008 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 22:50:18.352494 16008 rpc_server.cc:307] RPC server started. Bound to: 127.14.147.254:46225
I20260929 22:50:18.352561 16085 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.147.254:46225 every 8 connection(s)
I20260929 22:50:18.352880 16008 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790722213368738-14927-0/minicluster-data/master-0/data/info.pb
I20260929 22:50:18.353547 16086 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 22:50:18.355983 16086 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d20dfacee9bd43d3b147457b590bc600: Bootstrap starting.
I20260929 22:50:18.356587 16086 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d20dfacee9bd43d3b147457b590bc600: Neither blocks nor log segments found. Creating new log.
I20260929 22:50:18.356864 16086 log.cc:824] T 00000000000000000000000000000000 P d20dfacee9bd43d3b147457b590bc600: Log is configured to *not* fsync() on all Append() calls
I20260929 22:50:18.357556 16086 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d20dfacee9bd43d3b147457b590bc600: No bootstrap required, opened a new log
I20260929 22:50:18.359790 16086 raft_consensus.cc:348] T 00000000000000000000000000000000 P d20dfacee9bd43d3b147457b590bc600 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d20dfacee9bd43d3b147457b590bc600" member_type: VOTER last_known_addr { host: "127.14.147.254" port: 46225 } }
I20260929 22:50:18.359954 16086 raft_consensus.cc:374] T 00000000000000000000000000000000 P d20dfacee9bd43d3b147457b590bc600 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 22:50:18.359977 16086 raft_consensus.cc:729] T 00000000000000000000000000000000 P d20dfacee9bd43d3b147457b590bc600 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d20dfacee9bd43d3b147457b590bc600, State: Initialized, Role: FOLLOWER
I20260929 22:50:18.360129 16086 consensus_queue.cc:260] T 00000000000000000000000000000000 P d20dfacee9bd43d3b147457b590bc600 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d20dfacee9bd43d3b147457b590bc600" member_type: VOTER last_known_addr { host: "127.14.147.254" port: 46225 } }
I20260929 22:50:18.360201 16086 raft_consensus.cc:388] T 00000000000000000000000000000000 P d20dfacee9bd43d3b147457b590bc600 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 22:50:18.360236 16086 raft_consensus.cc:482] T 00000000000000000000000000000000 P d20dfacee9bd43d3b147457b590bc600 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 22:50:18.360282 16086 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d20dfacee9bd43d3b147457b590bc600 [term 0 FOLLOWER]: Advancing to term 1
I20260929 22:50:18.360612 14927 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhWOCLk/build/release/bin/kudu as pid 16008
I20260929 22:50:18.360703 14927 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790722213368738-14927-0/minicluster-data/master-0/wal/instance
I20260929 22:50:18.360849 14927 external_mini_cluster.cc:1428] Setting key dc0c8921e1719f5bc3f08a253c7a3399
I20260929 22:50:18.360929 16086 raft_consensus.cc:504] T 00000000000000000000000000000000 P d20dfacee9bd43d3b147457b590bc600 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d20dfacee9bd43d3b147457b590bc600" member_type: VOTER last_known_addr { host: "127.14.147.254" port: 46225 } }
I20260929 22:50:18.361039 16086 leader_election.cc:302] T 00000000000000000000000000000000 P d20dfacee9bd43d3b147457b590bc600 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d20dfacee9bd43d3b147457b590bc600; no voters:
I20260929 22:50:18.361227 16086 leader_election.cc:288] T 00000000000000000000000000000000 P d20dfacee9bd43d3b147457b590bc600 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 22:50:18.361340 16089 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d20dfacee9bd43d3b147457b590bc600 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 22:50:18.361465 16089 raft_consensus.cc:686] T 00000000000000000000000000000000 P d20dfacee9bd43d3b147457b590bc600 [term 1 LEADER]: Becoming Leader. State: Replica: d20dfacee9bd43d3b147457b590bc600, State: Running, Role: LEADER
I20260929 22:50:18.361511 16086 sys_catalog.cc:564] T 00000000000000000000000000000000 P d20dfacee9bd43d3b147457b590bc600 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 22:50:18.361547 16089 consensus_queue.cc:237] T 00000000000000000000000000000000 P d20dfacee9bd43d3b147457b590bc600 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d20dfacee9bd43d3b147457b590bc600" member_type: VOTER last_known_addr { host: "127.14.147.254" port: 46225 } }
I20260929 22:50:18.361972 16091 sys_catalog.cc:455] T 00000000000000000000000000000000 P d20dfacee9bd43d3b147457b590bc600 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d20dfacee9bd43d3b147457b590bc600" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d20dfacee9bd43d3b147457b590bc600" member_type: VOTER last_known_addr { host: "127.14.147.254" port: 46225 } } }
I20260929 22:50:18.361972 16089 sys_catalog.cc:455] T 00000000000000000000000000000000 P d20dfacee9bd43d3b147457b590bc600 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d20dfacee9bd43d3b147457b590bc600. Latest consensus state: current_term: 1 leader_uuid: "d20dfacee9bd43d3b147457b590bc600" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d20dfacee9bd43d3b147457b590bc600" member_type: VOTER last_known_addr { host: "127.14.147.254" port: 46225 } } }
I20260929 22:50:18.362102 16091 sys_catalog.cc:458] T 00000000000000000000000000000000 P d20dfacee9bd43d3b147457b590bc600 [sys.catalog]: This master's current role is: LEADER
I20260929 22:50:18.362108 16089 sys_catalog.cc:458] T 00000000000000000000000000000000 P d20dfacee9bd43d3b147457b590bc600 [sys.catalog]: This master's current role is: LEADER
I20260929 22:50:18.362375 16100 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 22:50:18.362989 16100 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 22:50:18.364745 16100 catalog_manager.cc:1347] Generated new cluster ID: 1738a55c19354deca9ed6a1f25c24e38
I20260929 22:50:18.364801 16100 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 29 22:50:18 dist-test-slave-98mk krb5kdc[15995](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790722218, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.147.254@KRBTEST.COM
I20260929 22:50:18.368840 16100 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 22:50:18.369402 16100 catalog_manager.cc:1504] Loading token signing keys...
I20260929 22:50:18.373111 16093 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:50:18.361341 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54232 (local address 127.14.147.254:46225)
0929 22:50:18.361680 (+ 339us) server_negotiation.cc:206] Beginning negotiation
0929 22:50:18.361691 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0929 22:50:18.361728 (+ 37us) server_negotiation.cc:407] Connection header received
0929 22:50:18.361869 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 22:50:18.361877 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 22:50:18.361937 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 22:50:18.362091 (+ 154us) server_negotiation.cc:226] Negotiated authn=SASL
0929 22:50:18.363083 (+ 992us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:50:18.364486 (+ 1403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:50:18.365575 (+ 1089us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:50:18.366232 (+ 657us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:50:18.366632 (+ 400us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 22:50:18.369290 (+ 2658us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 22:50:18.369295 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 22:50:18.369295 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 22:50:18.369309 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0929 22:50:18.371288 (+ 1979us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 22:50:18.371777 (+ 489us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 22:50:18.371781 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 22:50:18.371781 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 22:50:18.371896 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 22:50:18.372089 (+ 193us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 22:50:18.372090 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 22:50:18.372090 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 22:50:18.372362 (+ 272us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 22:50:18.372504 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0929 22:50:18.372607 (+ 103us) server_negotiation.cc:299] Negotiation successful
0929 22:50:18.372865 (+ 258us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":286,"thread_start_us":110,"threads_started":1}
I20260929 22:50:18.380460 16100 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d20dfacee9bd43d3b147457b590bc600: Generated new TSK 0
I20260929 22:50:18.380699 16100 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.14.147.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.147.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.147.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790722213368738-14927-0/krb5kdc/kudu_127.14.147.193.keytab.
Entry for principal kudu/127.14.147.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790722213368738-14927-0/krb5kdc/kudu_127.14.147.193.keytab.
I20260929 22:50:18.411722 14927 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhWOCLk/build/release/bin/kudu
/tmp/dist-test-taskhWOCLk/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790722213368738-14927-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790722213368738-14927-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790722213368738-14927-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790722213368738-14927-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.147.193:0
--local_ip_for_outbound_sockets=127.14.147.193
--webserver_interface=127.14.147.193
--webserver_port=0
--tserver_master_addrs=127.14.147.254:46225
--keytab_file=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790722213368738-14927-0/krb5kdc/kudu_127.14.147.193.keytab
--principal=kudu/127.14.147.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.147.212:36535
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790722213368738-14927-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790722213368738-14927-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790722213368738-14927-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 22:50:18.497030 16112 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 22:50:18.497263 16112 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 22:50:18.497289 16112 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 22:50:18.498484 16112 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 22:50:18.498521 16112 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 22:50:18.498541 16112 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260929 22:50:18.498558 16112 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.147.193
I20260929 22:50:18.500049 16112 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.147.212:36535
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790722213368738-14927-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790722213368738-14927-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790722213368738-14927-0/krb5kdc/kudu_127.14.147.193.keytab
--principal=kudu/127.14.147.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.147.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790722213368738-14927-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.14.147.193
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.14.147.254:46225
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16112
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.147.193
--log_dir=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790722213368738-14927-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 21:58:54 UTC on ccycloud.quasar-xguoti.root.comops.site
build id 4681390
I20260929 22:50:18.500259 16112 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 22:50:18.500450 16112 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 22:50:18.505442 16120 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 22:50:18.505465 16117 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 22:50:18.505442 16118 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 22:50:18.505735 16112 server_base.cc:1034] running on GCE node
I20260929 22:50:18.506011 16112 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 22:50:18.506263 16112 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 22:50:18.507470 16112 hybrid_clock.cc:648] HybridClock initialized: now 1790722218507445 us; error 36 us; skew 500 ppm
Sep 29 22:50:18 dist-test-slave-98mk krb5kdc[15995](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790722218, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.147.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 22:50:18.514155 16112 init.cc:377] Logged in from keytab as kudu/127.14.147.193@KRBTEST.COM (short username kudu)
I20260929 22:50:18.514616 16112 webserver.cc:460] Webserver started at http://127.14.147.193:35929/ using document root <none> and password file <none>
I20260929 22:50:18.514829 16112 fs_manager.cc:359] Metadata directory not provided
I20260929 22:50:18.514879 16112 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 22:50:18.515017 16112 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 22:50:18.516145 16112 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790722213368738-14927-0/minicluster-data/ts-0/data/instance:
uuid: "0e77842f330e4c0c9797c75ddf7a706a"
format_stamp: "Formatted at 2026-09-29 22:50:18 on dist-test-slave-98mk"
server_key: "d7f3717642338d0903aacede8be793ba"
server_key_iv: "db335e6cd52ef2d69306727064465825"
server_key_version: "encryptionkey@0"
I20260929 22:50:18.516495 16112 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790722213368738-14927-0/minicluster-data/ts-0/wal/instance:
uuid: "0e77842f330e4c0c9797c75ddf7a706a"
format_stamp: "Formatted at 2026-09-29 22:50:18 on dist-test-slave-98mk"
server_key: "d7f3717642338d0903aacede8be793ba"
server_key_iv: "db335e6cd52ef2d69306727064465825"
server_key_version: "encryptionkey@0"
I20260929 22:50:18.517939 16112 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260929 22:50:18.518672 16127 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 22:50:18.518839 16112 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260929 22:50:18.518924 16112 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790722213368738-14927-0/minicluster-data/ts-0/data,/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790722213368738-14927-0/minicluster-data/ts-0/wal
uuid: "0e77842f330e4c0c9797c75ddf7a706a"
format_stamp: "Formatted at 2026-09-29 22:50:18 on dist-test-slave-98mk"
server_key: "d7f3717642338d0903aacede8be793ba"
server_key_iv: "db335e6cd52ef2d69306727064465825"
server_key_version: "encryptionkey@0"
I20260929 22:50:18.518989 16112 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790722213368738-14927-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790722213368738-14927-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790722213368738-14927-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 22:50:18.542023 16112 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 22:50:18.544081 16112 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 22:50:18.544234 16112 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 22:50:18.544746 16112 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 22:50:18.544788 16112 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 22:50:18.544821 16112 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 22:50:18.544837 16112 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 29 22:50:18 dist-test-slave-98mk krb5kdc[15995](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790722218, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.147.193@KRBTEST.COM for kudu/127.14.147.254@KRBTEST.COM
I20260929 22:50:18.552605 16112 rpc_server.cc:307] RPC server started. Bound to: 127.14.147.193:45367
I20260929 22:50:18.552860 16242 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.147.193:45367 every 8 connection(s)
I20260929 22:50:18.553037 16112 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790722213368738-14927-0/minicluster-data/ts-0/data/info.pb
I20260929 22:50:18.555156 16093 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:50:18.545209 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.147.193:35433 (local address 127.14.147.254:46225)
0929 22:50:18.545283 (+ 74us) server_negotiation.cc:206] Beginning negotiation
0929 22:50:18.545286 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 22:50:18.545456 (+ 170us) server_negotiation.cc:407] Connection header received
0929 22:50:18.545940 (+ 484us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 22:50:18.545942 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 22:50:18.545954 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 22:50:18.546023 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0929 22:50:18.546839 (+ 816us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:50:18.547722 (+ 883us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:50:18.549185 (+ 1463us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:50:18.549508 (+ 323us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:50:18.549592 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 22:50:18.552274 (+ 2682us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 22:50:18.552276 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 22:50:18.552277 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 22:50:18.552286 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0929 22:50:18.553975 (+ 1689us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 22:50:18.554402 (+ 427us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 22:50:18.554403 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 22:50:18.554403 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 22:50:18.554450 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 22:50:18.554651 (+ 201us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 22:50:18.554652 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 22:50:18.554652 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 22:50:18.554759 (+ 107us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 22:50:18.554803 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0929 22:50:18.555024 (+ 221us) server_negotiation.cc:299] Negotiation successful
0929 22:50:18.555090 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260929 22:50:18.555373 16136 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:50:18.545162 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.147.254:46225 (local address 127.14.147.193:35433)
0929 22:50:18.545326 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0929 22:50:18.545369 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0929 22:50:18.545849 (+ 480us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 22:50:18.546110 (+ 261us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 22:50:18.546114 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 22:50:18.546454 (+ 340us) client_negotiation.cc:189] Negotiated authn=SASL
0929 22:50:18.546733 (+ 279us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 22:50:18.546740 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 22:50:18.547941 (+ 1201us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 22:50:18.547943 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 22:50:18.549050 (+ 1107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 22:50:18.549056 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 22:50:18.549682 (+ 626us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 22:50:18.549683 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 22:50:18.549983 (+ 300us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 22:50:18.550541 (+ 558us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 22:50:18.550551 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0929 22:50:18.552138 (+ 1587us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 22:50:18.554093 (+ 1955us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 22:50:18.554101 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 22:50:18.554102 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 22:50:18.554327 (+ 225us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 22:50:18.554506 (+ 179us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 22:50:18.554506 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 22:50:18.554507 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 22:50:18.554602 (+ 95us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 22:50:18.554801 (+ 199us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 22:50:18.554804 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 22:50:18.554930 (+ 126us) client_negotiation.cc:769] Sending connection context
0929 22:50:18.555013 (+ 83us) client_negotiation.cc:240] Negotiation successful
0929 22:50:18.555151 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":81,"threads_started":1}
I20260929 22:50:18.555891 16243 heartbeater.cc:347] Connected to a master server at 127.14.147.254:46225
I20260929 22:50:18.555979 16243 heartbeater.cc:464] Registering TS with master...
I20260929 22:50:18.556173 16243 heartbeater.cc:511] Master 127.14.147.254:46225 requested a full tablet report, sending...
I20260929 22:50:18.556794 16039 ts_manager.cc:194] Registered new tserver with Master: 0e77842f330e4c0c9797c75ddf7a706a (127.14.147.193:45367)
I20260929 22:50:18.556948 14927 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhWOCLk/build/release/bin/kudu as pid 16112
I20260929 22:50:18.557065 14927 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790722213368738-14927-0/minicluster-data/ts-0/wal/instance
I20260929 22:50:18.557212 14927 external_mini_cluster.cc:1428] Setting key fdd95b5c6819a7232980e4f4a1cdb990
I20260929 22:50:18.557579 16039 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.147.193@KRBTEST.COM'} at 127.14.147.193:35433
I20260929 22:50:18.563076 16093 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:50:18.557008 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.147.193:57647 (local address 127.14.147.254:46225)
0929 22:50:18.557045 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0929 22:50:18.557047 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 22:50:18.557460 (+ 413us) server_negotiation.cc:407] Connection header received
0929 22:50:18.557533 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 22:50:18.557534 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 22:50:18.557545 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 22:50:18.557589 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0929 22:50:18.557963 (+ 374us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:50:18.558615 (+ 652us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:50:18.559548 (+ 933us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:50:18.559746 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:50:18.559799 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 22:50:18.560766 (+ 967us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 22:50:18.560768 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 22:50:18.560768 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 22:50:18.560775 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0929 22:50:18.561956 (+ 1181us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 22:50:18.562308 (+ 352us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 22:50:18.562309 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 22:50:18.562309 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 22:50:18.562363 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 22:50:18.562553 (+ 190us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 22:50:18.562554 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 22:50:18.562554 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 22:50:18.562689 (+ 135us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 22:50:18.562745 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0929 22:50:18.562896 (+ 151us) server_negotiation.cc:299] Negotiation successful
0929 22:50:18.562975 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260929 22:50:18.563097 16249 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:50:18.556972 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.147.254:46225 (local address 127.14.147.193:57647)
0929 22:50:18.557399 (+ 427us) negotiation.cc:107] Waiting for socket to connect
0929 22:50:18.557410 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0929 22:50:18.557472 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 22:50:18.557593 (+ 121us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 22:50:18.557593 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 22:50:18.557790 (+ 197us) client_negotiation.cc:189] Negotiated authn=SASL
0929 22:50:18.557869 (+ 79us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 22:50:18.557872 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 22:50:18.558729 (+ 857us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 22:50:18.558730 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 22:50:18.559471 (+ 741us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 22:50:18.559474 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 22:50:18.559808 (+ 334us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 22:50:18.559809 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 22:50:18.559885 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 22:50:18.560349 (+ 464us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 22:50:18.560354 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0929 22:50:18.560666 (+ 312us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 22:50:18.562031 (+ 1365us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 22:50:18.562032 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 22:50:18.562032 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 22:50:18.562244 (+ 212us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 22:50:18.562415 (+ 171us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 22:50:18.562415 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 22:50:18.562416 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 22:50:18.562500 (+ 84us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 22:50:18.562744 (+ 244us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 22:50:18.562745 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 22:50:18.562807 (+ 62us) client_negotiation.cc:769] Sending connection context
0929 22:50:18.562864 (+ 57us) client_negotiation.cc:240] Negotiation successful
0929 22:50:18.562977 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":399,"thread_start_us":73,"threads_started":1}
WARNING: no policy specified for kudu/127.14.147.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.147.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.147.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790722213368738-14927-0/krb5kdc/kudu_127.14.147.194.keytab.
Entry for principal kudu/127.14.147.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790722213368738-14927-0/krb5kdc/kudu_127.14.147.194.keytab.
I20260929 22:50:18.579870 14927 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhWOCLk/build/release/bin/kudu
/tmp/dist-test-taskhWOCLk/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790722213368738-14927-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790722213368738-14927-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790722213368738-14927-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790722213368738-14927-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.147.194:0
--local_ip_for_outbound_sockets=127.14.147.194
--webserver_interface=127.14.147.194
--webserver_port=0
--tserver_master_addrs=127.14.147.254:46225
--keytab_file=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790722213368738-14927-0/krb5kdc/kudu_127.14.147.194.keytab
--principal=kudu/127.14.147.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.147.212:36535
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790722213368738-14927-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790722213368738-14927-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790722213368738-14927-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 22:50:18.661551 16252 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 22:50:18.661808 16252 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 22:50:18.661835 16252 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 22:50:18.663120 16252 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 22:50:18.663166 16252 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 22:50:18.663187 16252 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260929 22:50:18.663203 16252 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.147.194
I20260929 22:50:18.664744 16252 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.147.212:36535
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790722213368738-14927-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790722213368738-14927-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790722213368738-14927-0/krb5kdc/kudu_127.14.147.194.keytab
--principal=kudu/127.14.147.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.147.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790722213368738-14927-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.14.147.194
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.14.147.254:46225
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16252
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.147.194
--log_dir=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790722213368738-14927-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 21:58:54 UTC on ccycloud.quasar-xguoti.root.comops.site
build id 4681390
I20260929 22:50:18.664950 16252 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 22:50:18.665140 16252 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 22:50:18.670061 16258 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 22:50:18.670120 16260 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 22:50:18.670101 16257 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 22:50:18.670332 16252 server_base.cc:1034] running on GCE node
I20260929 22:50:18.670528 16252 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 22:50:18.670745 16252 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 22:50:18.671912 16252 hybrid_clock.cc:648] HybridClock initialized: now 1790722218671881 us; error 35 us; skew 500 ppm
Sep 29 22:50:18 dist-test-slave-98mk krb5kdc[15995](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790722218, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.147.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 22:50:18.674896 16252 init.cc:377] Logged in from keytab as kudu/127.14.147.194@KRBTEST.COM (short username kudu)
I20260929 22:50:18.675388 16252 webserver.cc:460] Webserver started at http://127.14.147.194:43695/ using document root <none> and password file <none>
I20260929 22:50:18.675679 16252 fs_manager.cc:359] Metadata directory not provided
I20260929 22:50:18.675733 16252 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 22:50:18.675861 16252 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 22:50:18.676937 16252 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790722213368738-14927-0/minicluster-data/ts-1/data/instance:
uuid: "2bcc9256a4f84ef19cdb214bf660076f"
format_stamp: "Formatted at 2026-09-29 22:50:18 on dist-test-slave-98mk"
server_key: "ba15e2e885d226c22480593c9c5f9bac"
server_key_iv: "98ec15dc445edeb054f5d90ab7148eb2"
server_key_version: "encryptionkey@0"
I20260929 22:50:18.677269 16252 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790722213368738-14927-0/minicluster-data/ts-1/wal/instance:
uuid: "2bcc9256a4f84ef19cdb214bf660076f"
format_stamp: "Formatted at 2026-09-29 22:50:18 on dist-test-slave-98mk"
server_key: "ba15e2e885d226c22480593c9c5f9bac"
server_key_iv: "98ec15dc445edeb054f5d90ab7148eb2"
server_key_version: "encryptionkey@0"
I20260929 22:50:18.678567 16252 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260929 22:50:18.679298 16267 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 22:50:18.679484 16252 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260929 22:50:18.679579 16252 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790722213368738-14927-0/minicluster-data/ts-1/data,/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790722213368738-14927-0/minicluster-data/ts-1/wal
uuid: "2bcc9256a4f84ef19cdb214bf660076f"
format_stamp: "Formatted at 2026-09-29 22:50:18 on dist-test-slave-98mk"
server_key: "ba15e2e885d226c22480593c9c5f9bac"
server_key_iv: "98ec15dc445edeb054f5d90ab7148eb2"
server_key_version: "encryptionkey@0"
I20260929 22:50:18.679648 16252 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790722213368738-14927-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790722213368738-14927-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790722213368738-14927-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 22:50:18.690685 16252 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 22:50:18.692704 16252 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 22:50:18.692842 16252 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 22:50:18.693332 16252 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 22:50:18.693373 16252 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 22:50:18.693404 16252 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 22:50:18.693427 16252 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 29 22:50:18 dist-test-slave-98mk krb5kdc[15995](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790722218, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.147.194@KRBTEST.COM for kudu/127.14.147.254@KRBTEST.COM
I20260929 22:50:18.700476 16252 rpc_server.cc:307] RPC server started. Bound to: 127.14.147.194:46111
I20260929 22:50:18.700589 16382 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.147.194:46111 every 8 connection(s)
I20260929 22:50:18.701160 16252 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790722213368738-14927-0/minicluster-data/ts-1/data/info.pb
I20260929 22:50:18.703562 16093 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:50:18.693812 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.147.194:37881 (local address 127.14.147.254:46225)
0929 22:50:18.693860 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0929 22:50:18.693862 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 22:50:18.694091 (+ 229us) server_negotiation.cc:407] Connection header received
0929 22:50:18.694655 (+ 564us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 22:50:18.694656 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 22:50:18.694669 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 22:50:18.694726 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0929 22:50:18.695675 (+ 949us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:50:18.696195 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:50:18.697719 (+ 1524us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:50:18.697933 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:50:18.697995 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 22:50:18.700704 (+ 2709us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 22:50:18.700705 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 22:50:18.700706 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 22:50:18.700715 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0929 22:50:18.702160 (+ 1445us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 22:50:18.702583 (+ 423us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 22:50:18.702583 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 22:50:18.702584 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 22:50:18.702640 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 22:50:18.702886 (+ 246us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 22:50:18.702887 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 22:50:18.702887 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 22:50:18.702989 (+ 102us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 22:50:18.703034 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0929 22:50:18.703362 (+ 328us) server_negotiation.cc:299] Negotiation successful
0929 22:50:18.703479 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260929 22:50:18.703693 16276 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:50:18.693814 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.147.254:46225 (local address 127.14.147.194:37881)
0929 22:50:18.693972 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0929 22:50:18.694008 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0929 22:50:18.694541 (+ 533us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 22:50:18.694846 (+ 305us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 22:50:18.694852 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 22:50:18.695228 (+ 376us) client_negotiation.cc:189] Negotiated authn=SASL
0929 22:50:18.695571 (+ 343us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 22:50:18.695578 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 22:50:18.696277 (+ 699us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 22:50:18.696278 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 22:50:18.697634 (+ 1356us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 22:50:18.697637 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 22:50:18.698018 (+ 381us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 22:50:18.698019 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 22:50:18.698203 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 22:50:18.698776 (+ 573us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 22:50:18.698787 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0929 22:50:18.700593 (+ 1806us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 22:50:18.702261 (+ 1668us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 22:50:18.702268 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 22:50:18.702270 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 22:50:18.702516 (+ 246us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 22:50:18.702715 (+ 199us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 22:50:18.702716 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 22:50:18.702716 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 22:50:18.702829 (+ 113us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 22:50:18.703064 (+ 235us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 22:50:18.703070 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 22:50:18.703200 (+ 130us) client_negotiation.cc:769] Sending connection context
0929 22:50:18.703302 (+ 102us) client_negotiation.cc:240] Negotiation successful
0929 22:50:18.703484 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":88,"threads_started":1}
I20260929 22:50:18.704170 16383 heartbeater.cc:347] Connected to a master server at 127.14.147.254:46225
I20260929 22:50:18.704260 16383 heartbeater.cc:464] Registering TS with master...
I20260929 22:50:18.704455 16383 heartbeater.cc:511] Master 127.14.147.254:46225 requested a full tablet report, sending...
I20260929 22:50:18.704809 16039 ts_manager.cc:194] Registered new tserver with Master: 2bcc9256a4f84ef19cdb214bf660076f (127.14.147.194:46111)
I20260929 22:50:18.704816 14927 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhWOCLk/build/release/bin/kudu as pid 16252
I20260929 22:50:18.704890 14927 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790722213368738-14927-0/minicluster-data/ts-1/wal/instance
I20260929 22:50:18.705075 14927 external_mini_cluster.cc:1428] Setting key 903fc8c2aff80ce80eaa7316b675b186
I20260929 22:50:18.705225 16039 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.147.194@KRBTEST.COM'} at 127.14.147.194:37881
I20260929 22:50:18.711088 16093 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:50:18.705293 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.147.194:55471 (local address 127.14.147.254:46225)
0929 22:50:18.705329 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0929 22:50:18.705331 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 22:50:18.705474 (+ 143us) server_negotiation.cc:407] Connection header received
0929 22:50:18.705516 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 22:50:18.705517 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 22:50:18.705527 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 22:50:18.705555 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0929 22:50:18.706362 (+ 807us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:50:18.706910 (+ 548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:50:18.707648 (+ 738us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:50:18.707899 (+ 251us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:50:18.707960 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 22:50:18.708840 (+ 880us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 22:50:18.708841 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 22:50:18.708842 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 22:50:18.708849 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0929 22:50:18.710038 (+ 1189us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 22:50:18.710333 (+ 295us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 22:50:18.710334 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 22:50:18.710335 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 22:50:18.710389 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 22:50:18.710549 (+ 160us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 22:50:18.710550 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 22:50:18.710550 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 22:50:18.710698 (+ 148us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 22:50:18.710755 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0929 22:50:18.710909 (+ 154us) server_negotiation.cc:299] Negotiation successful
0929 22:50:18.710992 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260929 22:50:18.711105 16389 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:50:18.705244 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.147.254:46225 (local address 127.14.147.194:55471)
0929 22:50:18.705393 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0929 22:50:18.705409 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 22:50:18.705470 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 22:50:18.706066 (+ 596us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 22:50:18.706067 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 22:50:18.706223 (+ 156us) client_negotiation.cc:189] Negotiated authn=SASL
0929 22:50:18.706287 (+ 64us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 22:50:18.706288 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 22:50:18.707003 (+ 715us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 22:50:18.707003 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 22:50:18.707581 (+ 578us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 22:50:18.707583 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 22:50:18.707972 (+ 389us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 22:50:18.707973 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 22:50:18.708053 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 22:50:18.708458 (+ 405us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 22:50:18.708463 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0929 22:50:18.708761 (+ 298us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 22:50:18.710113 (+ 1352us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 22:50:18.710114 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 22:50:18.710114 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 22:50:18.710274 (+ 160us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 22:50:18.710441 (+ 167us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 22:50:18.710441 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 22:50:18.710441 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 22:50:18.710499 (+ 58us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 22:50:18.710754 (+ 255us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 22:50:18.710755 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 22:50:18.710814 (+ 59us) client_negotiation.cc:769] Sending connection context
0929 22:50:18.710880 (+ 66us) client_negotiation.cc:240] Negotiation successful
0929 22:50:18.710992 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":74,"threads_started":1}
WARNING: no policy specified for kudu/127.14.147.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.147.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.147.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790722213368738-14927-0/krb5kdc/kudu_127.14.147.195.keytab.
Entry for principal kudu/127.14.147.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790722213368738-14927-0/krb5kdc/kudu_127.14.147.195.keytab.
I20260929 22:50:18.727232 14927 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhWOCLk/build/release/bin/kudu
/tmp/dist-test-taskhWOCLk/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790722213368738-14927-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790722213368738-14927-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790722213368738-14927-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790722213368738-14927-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.147.195:0
--local_ip_for_outbound_sockets=127.14.147.195
--webserver_interface=127.14.147.195
--webserver_port=0
--tserver_master_addrs=127.14.147.254:46225
--keytab_file=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790722213368738-14927-0/krb5kdc/kudu_127.14.147.195.keytab
--principal=kudu/127.14.147.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.147.212:36535
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790722213368738-14927-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790722213368738-14927-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790722213368738-14927-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 22:50:18.808171 16392 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 22:50:18.808389 16392 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 22:50:18.808415 16392 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 22:50:18.809624 16392 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 22:50:18.809667 16392 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 22:50:18.809687 16392 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260929 22:50:18.809705 16392 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.147.195
I20260929 22:50:18.811208 16392 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.147.212:36535
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790722213368738-14927-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790722213368738-14927-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790722213368738-14927-0/krb5kdc/kudu_127.14.147.195.keytab
--principal=kudu/127.14.147.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.147.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790722213368738-14927-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.14.147.195
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.14.147.254:46225
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16392
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.147.195
--log_dir=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790722213368738-14927-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 21:58:54 UTC on ccycloud.quasar-xguoti.root.comops.site
build id 4681390
I20260929 22:50:18.811442 16392 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 22:50:18.811657 16392 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 22:50:18.816496 16397 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 22:50:18.816624 16392 server_base.cc:1034] running on GCE node
W20260929 22:50:18.816534 16400 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 22:50:18.816560 16398 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 22:50:18.816952 16392 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 22:50:18.817193 16392 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 22:50:18.818342 16392 hybrid_clock.cc:648] HybridClock initialized: now 1790722218818332 us; error 31 us; skew 500 ppm
Sep 29 22:50:18 dist-test-slave-98mk krb5kdc[15995](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790722218, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.147.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 22:50:18.820920 16392 init.cc:377] Logged in from keytab as kudu/127.14.147.195@KRBTEST.COM (short username kudu)
I20260929 22:50:18.821477 16392 webserver.cc:460] Webserver started at http://127.14.147.195:37061/ using document root <none> and password file <none>
I20260929 22:50:18.821708 16392 fs_manager.cc:359] Metadata directory not provided
I20260929 22:50:18.821758 16392 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 22:50:18.821888 16392 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 22:50:18.822983 16392 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790722213368738-14927-0/minicluster-data/ts-2/data/instance:
uuid: "3a977c79b99544e08ef536df3e5fef57"
format_stamp: "Formatted at 2026-09-29 22:50:18 on dist-test-slave-98mk"
server_key: "207ec9531888e15fb989a4cd2d020693"
server_key_iv: "0453b9f2c46dc199ab45e9e9406d92cc"
server_key_version: "encryptionkey@0"
I20260929 22:50:18.823330 16392 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790722213368738-14927-0/minicluster-data/ts-2/wal/instance:
uuid: "3a977c79b99544e08ef536df3e5fef57"
format_stamp: "Formatted at 2026-09-29 22:50:18 on dist-test-slave-98mk"
server_key: "207ec9531888e15fb989a4cd2d020693"
server_key_iv: "0453b9f2c46dc199ab45e9e9406d92cc"
server_key_version: "encryptionkey@0"
I20260929 22:50:18.824718 16392 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260929 22:50:18.825568 16407 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 22:50:18.825758 16392 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260929 22:50:18.825845 16392 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790722213368738-14927-0/minicluster-data/ts-2/data,/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790722213368738-14927-0/minicluster-data/ts-2/wal
uuid: "3a977c79b99544e08ef536df3e5fef57"
format_stamp: "Formatted at 2026-09-29 22:50:18 on dist-test-slave-98mk"
server_key: "207ec9531888e15fb989a4cd2d020693"
server_key_iv: "0453b9f2c46dc199ab45e9e9406d92cc"
server_key_version: "encryptionkey@0"
I20260929 22:50:18.825917 16392 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790722213368738-14927-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790722213368738-14927-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790722213368738-14927-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 22:50:18.846940 16392 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 22:50:18.848735 16392 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 22:50:18.848870 16392 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 22:50:18.849371 16392 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 22:50:18.849416 16392 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 22:50:18.849444 16392 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 22:50:18.849469 16392 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 29 22:50:18 dist-test-slave-98mk krb5kdc[15995](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790722218, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.147.195@KRBTEST.COM for kudu/127.14.147.254@KRBTEST.COM
I20260929 22:50:18.856374 16392 rpc_server.cc:307] RPC server started. Bound to: 127.14.147.195:44731
I20260929 22:50:18.856446 16522 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.147.195:44731 every 8 connection(s)
I20260929 22:50:18.856776 16392 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790722213368738-14927-0/minicluster-data/ts-2/data/info.pb
I20260929 22:50:18.859117 16093 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:50:18.849819 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.147.195:41677 (local address 127.14.147.254:46225)
0929 22:50:18.849872 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0929 22:50:18.849874 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 22:50:18.850134 (+ 260us) server_negotiation.cc:407] Connection header received
0929 22:50:18.850668 (+ 534us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 22:50:18.850670 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 22:50:18.850685 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 22:50:18.850748 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0929 22:50:18.851568 (+ 820us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:50:18.852338 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:50:18.853405 (+ 1067us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:50:18.853620 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:50:18.853678 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 22:50:18.856403 (+ 2725us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 22:50:18.856405 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 22:50:18.856405 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 22:50:18.856413 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0929 22:50:18.857844 (+ 1431us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 22:50:18.858226 (+ 382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 22:50:18.858228 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 22:50:18.858228 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 22:50:18.858286 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 22:50:18.858498 (+ 212us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 22:50:18.858499 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 22:50:18.858500 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 22:50:18.858620 (+ 120us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 22:50:18.858678 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0929 22:50:18.858942 (+ 264us) server_negotiation.cc:299] Negotiation successful
0929 22:50:18.859026 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260929 22:50:18.859287 16416 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:50:18.849821 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.147.254:46225 (local address 127.14.147.195:41677)
0929 22:50:18.849992 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0929 22:50:18.850046 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0929 22:50:18.850563 (+ 517us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 22:50:18.850817 (+ 254us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 22:50:18.850821 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 22:50:18.851165 (+ 344us) client_negotiation.cc:189] Negotiated authn=SASL
0929 22:50:18.851471 (+ 306us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 22:50:18.851477 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 22:50:18.852445 (+ 968us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 22:50:18.852446 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 22:50:18.853313 (+ 867us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 22:50:18.853315 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 22:50:18.853709 (+ 394us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 22:50:18.853710 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 22:50:18.853825 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 22:50:18.854352 (+ 527us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 22:50:18.854362 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0929 22:50:18.856308 (+ 1946us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 22:50:18.857927 (+ 1619us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 22:50:18.857935 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 22:50:18.857937 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 22:50:18.858157 (+ 220us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 22:50:18.858340 (+ 183us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 22:50:18.858340 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 22:50:18.858341 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 22:50:18.858445 (+ 104us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 22:50:18.858677 (+ 232us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 22:50:18.858683 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 22:50:18.858821 (+ 138us) client_negotiation.cc:769] Sending connection context
0929 22:50:18.858912 (+ 91us) client_negotiation.cc:240] Negotiation successful
0929 22:50:18.859072 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":85,"threads_started":1}
I20260929 22:50:18.859800 16523 heartbeater.cc:347] Connected to a master server at 127.14.147.254:46225
I20260929 22:50:18.859879 16523 heartbeater.cc:464] Registering TS with master...
I20260929 22:50:18.860047 16523 heartbeater.cc:511] Master 127.14.147.254:46225 requested a full tablet report, sending...
I20260929 22:50:18.860363 16040 ts_manager.cc:194] Registered new tserver with Master: 3a977c79b99544e08ef536df3e5fef57 (127.14.147.195:44731)
I20260929 22:50:18.860755 16040 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.147.195@KRBTEST.COM'} at 127.14.147.195:41677
I20260929 22:50:18.862643 14927 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhWOCLk/build/release/bin/kudu as pid 16392
I20260929 22:50:18.862730 14927 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790722213368738-14927-0/minicluster-data/ts-2/wal/instance
I20260929 22:50:18.862907 14927 external_mini_cluster.cc:1428] Setting key 0a54e37932a2cb7593a38ee707282cb9
I20260929 22:50:18.863984 14927 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 22:50:18.866233 16093 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:50:18.860824 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.147.195:42557 (local address 127.14.147.254:46225)
0929 22:50:18.860944 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0929 22:50:18.860946 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 22:50:18.860953 (+ 7us) server_negotiation.cc:407] Connection header received
0929 22:50:18.860970 (+ 17us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 22:50:18.860971 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 22:50:18.860980 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 22:50:18.861020 (+ 40us) server_negotiation.cc:226] Negotiated authn=SASL
0929 22:50:18.861340 (+ 320us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:50:18.861805 (+ 465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:50:18.862572 (+ 767us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:50:18.862793 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:50:18.862857 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 22:50:18.863924 (+ 1067us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 22:50:18.863925 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 22:50:18.863926 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 22:50:18.863933 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0929 22:50:18.865232 (+ 1299us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 22:50:18.865535 (+ 303us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 22:50:18.865536 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 22:50:18.865537 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 22:50:18.865594 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 22:50:18.865753 (+ 159us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 22:50:18.865754 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 22:50:18.865755 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 22:50:18.865870 (+ 115us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 22:50:18.865929 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0929 22:50:18.866066 (+ 137us) server_negotiation.cc:299] Negotiation successful
0929 22:50:18.866139 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":101}
I20260929 22:50:18.866247 16529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:50:18.860747 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.147.254:46225 (local address 127.14.147.195:42557)
0929 22:50:18.860871 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0929 22:50:18.860882 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0929 22:50:18.860927 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 22:50:18.861024 (+ 97us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 22:50:18.861025 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 22:50:18.861216 (+ 191us) client_negotiation.cc:189] Negotiated authn=SASL
0929 22:50:18.861272 (+ 56us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 22:50:18.861274 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 22:50:18.861896 (+ 622us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 22:50:18.861897 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 22:50:18.862500 (+ 603us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 22:50:18.862502 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 22:50:18.862859 (+ 357us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 22:50:18.862860 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 22:50:18.862937 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 22:50:18.863450 (+ 513us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 22:50:18.863456 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0929 22:50:18.863830 (+ 374us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 22:50:18.865314 (+ 1484us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 22:50:18.865315 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 22:50:18.865315 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 22:50:18.865473 (+ 158us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 22:50:18.865645 (+ 172us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 22:50:18.865646 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 22:50:18.865646 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 22:50:18.865691 (+ 45us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 22:50:18.865927 (+ 236us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 22:50:18.865928 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 22:50:18.865976 (+ 48us) client_negotiation.cc:769] Sending connection context
0929 22:50:18.866038 (+ 62us) client_negotiation.cc:240] Negotiation successful
0929 22:50:18.866141 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":96,"thread_start_us":68,"threads_started":1}
Sep 29 22:50:18 dist-test-slave-98mk krb5kdc[15995](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790722218, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 29 22:50:18 dist-test-slave-98mk krb5kdc[15995](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790722218, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.14.147.254@KRBTEST.COM
I20260929 22:50:18.886696 16093 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:50:18.879330 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54240 (local address 127.14.147.254:46225)
0929 22:50:18.879398 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0929 22:50:18.879400 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 22:50:18.879408 (+ 8us) server_negotiation.cc:407] Connection header received
0929 22:50:18.879461 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 22:50:18.879462 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 22:50:18.879474 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 22:50:18.879523 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0929 22:50:18.880025 (+ 502us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:50:18.880625 (+ 600us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:50:18.881369 (+ 744us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:50:18.881569 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:50:18.881633 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 22:50:18.883907 (+ 2274us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 22:50:18.883908 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 22:50:18.883909 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 22:50:18.883917 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0929 22:50:18.885339 (+ 1422us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 22:50:18.885736 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 22:50:18.885737 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 22:50:18.885737 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 22:50:18.885804 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 22:50:18.886024 (+ 220us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 22:50:18.886025 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 22:50:18.886026 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 22:50:18.886183 (+ 157us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 22:50:18.886295 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0929 22:50:18.886489 (+ 194us) server_negotiation.cc:299] Negotiation successful
0929 22:50:18.886584 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260929 22:50:18.888552 16040 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:54240:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 22:50:18.890125 16040 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 22:50:18.896595 16544 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:50:18.892751 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.147.195:44731 (local address 127.0.0.1:50490)
0929 22:50:18.893018 (+ 267us) negotiation.cc:107] Waiting for socket to connect
0929 22:50:18.893028 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0929 22:50:18.893079 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 22:50:18.893322 (+ 243us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 22:50:18.893323 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 22:50:18.893325 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 22:50:18.893383 (+ 58us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 22:50:18.893385 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 22:50:18.894246 (+ 861us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 22:50:18.894247 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 22:50:18.895455 (+ 1208us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 22:50:18.895458 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 22:50:18.896263 (+ 805us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 22:50:18.896264 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 22:50:18.896360 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 22:50:18.896377 (+ 17us) client_negotiation.cc:769] Sending connection context
0929 22:50:18.896451 (+ 74us) client_negotiation.cc:240] Negotiation successful
0929 22:50:18.896512 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":246,"thread_start_us":61,"threads_started":1}
I20260929 22:50:18.896628 16545 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:50:18.892866 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50490 (local address 127.14.147.195:44731)
0929 22:50:18.893084 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0929 22:50:18.893090 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 22:50:18.893101 (+ 11us) server_negotiation.cc:407] Connection header received
0929 22:50:18.893164 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 22:50:18.893166 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 22:50:18.893181 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 22:50:18.893252 (+ 71us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 22:50:18.893449 (+ 197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:50:18.894166 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:50:18.895527 (+ 1361us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:50:18.896187 (+ 660us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:50:18.896249 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 22:50:18.896290 (+ 41us) server_negotiation.cc:1035] Waiting for connection context
0929 22:50:18.896470 (+ 180us) server_negotiation.cc:299] Negotiation successful
0929 22:50:18.896509 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":116,"thread_start_us":70,"threads_started":1}
I20260929 22:50:18.896694 16541 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:50:18.892569 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.147.194:46111 (local address 127.0.0.1:54288)
0929 22:50:18.892932 (+ 363us) negotiation.cc:107] Waiting for socket to connect
0929 22:50:18.892940 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0929 22:50:18.892996 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 22:50:18.893281 (+ 285us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 22:50:18.893284 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 22:50:18.893286 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 22:50:18.893374 (+ 88us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 22:50:18.893376 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 22:50:18.894697 (+ 1321us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 22:50:18.894698 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 22:50:18.895759 (+ 1061us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 22:50:18.895761 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 22:50:18.896469 (+ 708us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 22:50:18.896470 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 22:50:18.896531 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 22:50:18.896535 (+ 4us) client_negotiation.cc:769] Sending connection context
0929 22:50:18.896588 (+ 53us) client_negotiation.cc:240] Negotiation successful
0929 22:50:18.896631 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":347,"mutex_wait_us":36,"thread_start_us":92,"threads_started":1}
I20260929 22:50:18.896912 16542 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:50:18.892715 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54288 (local address 127.14.147.194:46111)
0929 22:50:18.892887 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0929 22:50:18.892893 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 22:50:18.893079 (+ 186us) server_negotiation.cc:407] Connection header received
0929 22:50:18.893124 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 22:50:18.893127 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 22:50:18.893143 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 22:50:18.893207 (+ 64us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 22:50:18.893452 (+ 245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:50:18.894594 (+ 1142us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:50:18.895848 (+ 1254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:50:18.896388 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:50:18.896461 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 22:50:18.896513 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0929 22:50:18.896774 (+ 261us) server_negotiation.cc:299] Negotiation successful
0929 22:50:18.896822 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":144,"thread_start_us":72,"threads_started":1}
I20260929 22:50:18.897421 16317 tablet_service.cc:1467] Processing CreateTablet for tablet b327bea83641424e8de6b692c1ef57b9 (DEFAULT_TABLE table=test-table [id=7b75cc67d50343d589d7038727326749]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 22:50:18.897392 16457 tablet_service.cc:1467] Processing CreateTablet for tablet b327bea83641424e8de6b692c1ef57b9 (DEFAULT_TABLE table=test-table [id=7b75cc67d50343d589d7038727326749]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 22:50:18.897766 16317 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b327bea83641424e8de6b692c1ef57b9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 22:50:18.897766 16457 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b327bea83641424e8de6b692c1ef57b9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 22:50:18.897864 16540 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:50:18.892554 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.147.193:45367 (local address 127.0.0.1:59622)
0929 22:50:18.892881 (+ 327us) negotiation.cc:107] Waiting for socket to connect
0929 22:50:18.892928 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0929 22:50:18.892996 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 22:50:18.893427 (+ 431us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 22:50:18.893428 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 22:50:18.893431 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 22:50:18.893463 (+ 32us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 22:50:18.893464 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 22:50:18.894694 (+ 1230us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 22:50:18.894695 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 22:50:18.895774 (+ 1079us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 22:50:18.895776 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 22:50:18.897513 (+ 1737us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 22:50:18.897514 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 22:50:18.897578 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 22:50:18.897582 (+ 4us) client_negotiation.cc:769] Sending connection context
0929 22:50:18.897711 (+ 129us) client_negotiation.cc:240] Negotiation successful
0929 22:50:18.897755 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":299,"mutex_wait_us":26,"thread_start_us":44,"threads_started":1}
I20260929 22:50:18.897985 16543 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:50:18.892718 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59622 (local address 127.14.147.193:45367)
0929 22:50:18.893249 (+ 531us) server_negotiation.cc:206] Beginning negotiation
0929 22:50:18.893253 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 22:50:18.893262 (+ 9us) server_negotiation.cc:407] Connection header received
0929 22:50:18.893295 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 22:50:18.893297 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 22:50:18.893310 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 22:50:18.893374 (+ 64us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 22:50:18.893532 (+ 158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:50:18.894614 (+ 1082us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:50:18.896552 (+ 1938us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:50:18.897442 (+ 890us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:50:18.897510 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 22:50:18.897560 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0929 22:50:18.897746 (+ 186us) server_negotiation.cc:299] Negotiation successful
0929 22:50:18.897789 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":495,"thread_start_us":77,"threads_started":1}
I20260929 22:50:18.900185 16175 tablet_service.cc:1467] Processing CreateTablet for tablet b327bea83641424e8de6b692c1ef57b9 (DEFAULT_TABLE table=test-table [id=7b75cc67d50343d589d7038727326749]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 22:50:18.900520 16175 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b327bea83641424e8de6b692c1ef57b9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 22:50:18.901242 16546 tablet_bootstrap.cc:492] T b327bea83641424e8de6b692c1ef57b9 P 3a977c79b99544e08ef536df3e5fef57: Bootstrap starting.
I20260929 22:50:18.901670 16547 tablet_bootstrap.cc:492] T b327bea83641424e8de6b692c1ef57b9 P 2bcc9256a4f84ef19cdb214bf660076f: Bootstrap starting.
I20260929 22:50:18.902258 16547 tablet_bootstrap.cc:654] T b327bea83641424e8de6b692c1ef57b9 P 2bcc9256a4f84ef19cdb214bf660076f: Neither blocks nor log segments found. Creating new log.
I20260929 22:50:18.902294 16546 tablet_bootstrap.cc:654] T b327bea83641424e8de6b692c1ef57b9 P 3a977c79b99544e08ef536df3e5fef57: Neither blocks nor log segments found. Creating new log.
I20260929 22:50:18.902554 16547 log.cc:824] T b327bea83641424e8de6b692c1ef57b9 P 2bcc9256a4f84ef19cdb214bf660076f: Log is configured to *not* fsync() on all Append() calls
I20260929 22:50:18.902621 16546 log.cc:824] T b327bea83641424e8de6b692c1ef57b9 P 3a977c79b99544e08ef536df3e5fef57: Log is configured to *not* fsync() on all Append() calls
I20260929 22:50:18.903209 16547 tablet_bootstrap.cc:492] T b327bea83641424e8de6b692c1ef57b9 P 2bcc9256a4f84ef19cdb214bf660076f: No bootstrap required, opened a new log
I20260929 22:50:18.903270 16547 ts_tablet_manager.cc:1397] T b327bea83641424e8de6b692c1ef57b9 P 2bcc9256a4f84ef19cdb214bf660076f: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260929 22:50:18.903385 16546 tablet_bootstrap.cc:492] T b327bea83641424e8de6b692c1ef57b9 P 3a977c79b99544e08ef536df3e5fef57: No bootstrap required, opened a new log
I20260929 22:50:18.903498 16546 ts_tablet_manager.cc:1397] T b327bea83641424e8de6b692c1ef57b9 P 3a977c79b99544e08ef536df3e5fef57: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260929 22:50:18.904093 16550 tablet_bootstrap.cc:492] T b327bea83641424e8de6b692c1ef57b9 P 0e77842f330e4c0c9797c75ddf7a706a: Bootstrap starting.
I20260929 22:50:18.904794 16550 tablet_bootstrap.cc:654] T b327bea83641424e8de6b692c1ef57b9 P 0e77842f330e4c0c9797c75ddf7a706a: Neither blocks nor log segments found. Creating new log.
I20260929 22:50:18.905192 16550 log.cc:824] T b327bea83641424e8de6b692c1ef57b9 P 0e77842f330e4c0c9797c75ddf7a706a: Log is configured to *not* fsync() on all Append() calls
I20260929 22:50:18.905226 16547 raft_consensus.cc:348] T b327bea83641424e8de6b692c1ef57b9 P 2bcc9256a4f84ef19cdb214bf660076f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e77842f330e4c0c9797c75ddf7a706a" member_type: VOTER last_known_addr { host: "127.14.147.193" port: 45367 } } peers { permanent_uuid: "2bcc9256a4f84ef19cdb214bf660076f" member_type: VOTER last_known_addr { host: "127.14.147.194" port: 46111 } } peers { permanent_uuid: "3a977c79b99544e08ef536df3e5fef57" member_type: VOTER last_known_addr { host: "127.14.147.195" port: 44731 } }
I20260929 22:50:18.905280 16546 raft_consensus.cc:348] T b327bea83641424e8de6b692c1ef57b9 P 3a977c79b99544e08ef536df3e5fef57 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e77842f330e4c0c9797c75ddf7a706a" member_type: VOTER last_known_addr { host: "127.14.147.193" port: 45367 } } peers { permanent_uuid: "2bcc9256a4f84ef19cdb214bf660076f" member_type: VOTER last_known_addr { host: "127.14.147.194" port: 46111 } } peers { permanent_uuid: "3a977c79b99544e08ef536df3e5fef57" member_type: VOTER last_known_addr { host: "127.14.147.195" port: 44731 } }
I20260929 22:50:18.905392 16547 raft_consensus.cc:374] T b327bea83641424e8de6b692c1ef57b9 P 2bcc9256a4f84ef19cdb214bf660076f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 22:50:18.905426 16547 raft_consensus.cc:729] T b327bea83641424e8de6b692c1ef57b9 P 2bcc9256a4f84ef19cdb214bf660076f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2bcc9256a4f84ef19cdb214bf660076f, State: Initialized, Role: FOLLOWER
I20260929 22:50:18.905428 16546 raft_consensus.cc:374] T b327bea83641424e8de6b692c1ef57b9 P 3a977c79b99544e08ef536df3e5fef57 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 22:50:18.905471 16546 raft_consensus.cc:729] T b327bea83641424e8de6b692c1ef57b9 P 3a977c79b99544e08ef536df3e5fef57 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3a977c79b99544e08ef536df3e5fef57, State: Initialized, Role: FOLLOWER
I20260929 22:50:18.905550 16547 consensus_queue.cc:260] T b327bea83641424e8de6b692c1ef57b9 P 2bcc9256a4f84ef19cdb214bf660076f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e77842f330e4c0c9797c75ddf7a706a" member_type: VOTER last_known_addr { host: "127.14.147.193" port: 45367 } } peers { permanent_uuid: "2bcc9256a4f84ef19cdb214bf660076f" member_type: VOTER last_known_addr { host: "127.14.147.194" port: 46111 } } peers { permanent_uuid: "3a977c79b99544e08ef536df3e5fef57" member_type: VOTER last_known_addr { host: "127.14.147.195" port: 44731 } }
I20260929 22:50:18.905596 16546 consensus_queue.cc:260] T b327bea83641424e8de6b692c1ef57b9 P 3a977c79b99544e08ef536df3e5fef57 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e77842f330e4c0c9797c75ddf7a706a" member_type: VOTER last_known_addr { host: "127.14.147.193" port: 45367 } } peers { permanent_uuid: "2bcc9256a4f84ef19cdb214bf660076f" member_type: VOTER last_known_addr { host: "127.14.147.194" port: 46111 } } peers { permanent_uuid: "3a977c79b99544e08ef536df3e5fef57" member_type: VOTER last_known_addr { host: "127.14.147.195" port: 44731 } }
I20260929 22:50:18.905799 16547 ts_tablet_manager.cc:1428] T b327bea83641424e8de6b692c1ef57b9 P 2bcc9256a4f84ef19cdb214bf660076f: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260929 22:50:18.905807 16383 heartbeater.cc:503] Master 127.14.147.254:46225 was elected leader, sending a full tablet report...
I20260929 22:50:18.905838 16546 ts_tablet_manager.cc:1428] T b327bea83641424e8de6b692c1ef57b9 P 3a977c79b99544e08ef536df3e5fef57: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260929 22:50:18.905951 16523 heartbeater.cc:503] Master 127.14.147.254:46225 was elected leader, sending a full tablet report...
I20260929 22:50:18.906678 16550 tablet_bootstrap.cc:492] T b327bea83641424e8de6b692c1ef57b9 P 0e77842f330e4c0c9797c75ddf7a706a: No bootstrap required, opened a new log
I20260929 22:50:18.906752 16550 ts_tablet_manager.cc:1397] T b327bea83641424e8de6b692c1ef57b9 P 0e77842f330e4c0c9797c75ddf7a706a: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260929 22:50:18.908290 16550 raft_consensus.cc:348] T b327bea83641424e8de6b692c1ef57b9 P 0e77842f330e4c0c9797c75ddf7a706a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e77842f330e4c0c9797c75ddf7a706a" member_type: VOTER last_known_addr { host: "127.14.147.193" port: 45367 } } peers { permanent_uuid: "2bcc9256a4f84ef19cdb214bf660076f" member_type: VOTER last_known_addr { host: "127.14.147.194" port: 46111 } } peers { permanent_uuid: "3a977c79b99544e08ef536df3e5fef57" member_type: VOTER last_known_addr { host: "127.14.147.195" port: 44731 } }
I20260929 22:50:18.908430 16550 raft_consensus.cc:374] T b327bea83641424e8de6b692c1ef57b9 P 0e77842f330e4c0c9797c75ddf7a706a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 22:50:18.908452 16550 raft_consensus.cc:729] T b327bea83641424e8de6b692c1ef57b9 P 0e77842f330e4c0c9797c75ddf7a706a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0e77842f330e4c0c9797c75ddf7a706a, State: Initialized, Role: FOLLOWER
I20260929 22:50:18.908543 16550 consensus_queue.cc:260] T b327bea83641424e8de6b692c1ef57b9 P 0e77842f330e4c0c9797c75ddf7a706a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e77842f330e4c0c9797c75ddf7a706a" member_type: VOTER last_known_addr { host: "127.14.147.193" port: 45367 } } peers { permanent_uuid: "2bcc9256a4f84ef19cdb214bf660076f" member_type: VOTER last_known_addr { host: "127.14.147.194" port: 46111 } } peers { permanent_uuid: "3a977c79b99544e08ef536df3e5fef57" member_type: VOTER last_known_addr { host: "127.14.147.195" port: 44731 } }
I20260929 22:50:18.908776 16550 ts_tablet_manager.cc:1428] T b327bea83641424e8de6b692c1ef57b9 P 0e77842f330e4c0c9797c75ddf7a706a: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260929 22:50:18.908928 16243 heartbeater.cc:503] Master 127.14.147.254:46225 was elected leader, sending a full tablet report...
W20260929 22:50:18.952041 16384 tablet.cc:2369] T b327bea83641424e8de6b692c1ef57b9 P 2bcc9256a4f84ef19cdb214bf660076f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 22:50:18.994186 16552 raft_consensus.cc:482] T b327bea83641424e8de6b692c1ef57b9 P 2bcc9256a4f84ef19cdb214bf660076f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 22:50:18.994336 16552 raft_consensus.cc:504] T b327bea83641424e8de6b692c1ef57b9 P 2bcc9256a4f84ef19cdb214bf660076f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e77842f330e4c0c9797c75ddf7a706a" member_type: VOTER last_known_addr { host: "127.14.147.193" port: 45367 } } peers { permanent_uuid: "2bcc9256a4f84ef19cdb214bf660076f" member_type: VOTER last_known_addr { host: "127.14.147.194" port: 46111 } } peers { permanent_uuid: "3a977c79b99544e08ef536df3e5fef57" member_type: VOTER last_known_addr { host: "127.14.147.195" port: 44731 } }
I20260929 22:50:18.994699 16552 leader_election.cc:288] T b327bea83641424e8de6b692c1ef57b9 P 2bcc9256a4f84ef19cdb214bf660076f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0e77842f330e4c0c9797c75ddf7a706a (127.14.147.193:45367), 3a977c79b99544e08ef536df3e5fef57 (127.14.147.195:44731)
I20260929 22:50:18.997658 16545 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:50:18.995052 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.147.194:39227 (local address 127.14.147.195:44731)
0929 22:50:18.995096 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0929 22:50:18.995098 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 22:50:18.995340 (+ 242us) server_negotiation.cc:407] Connection header received
0929 22:50:18.995437 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 22:50:18.995438 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 22:50:18.995448 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 22:50:18.995503 (+ 55us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 22:50:18.995643 (+ 140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:50:18.996099 (+ 456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:50:18.996980 (+ 881us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:50:18.997342 (+ 362us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:50:18.997404 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 22:50:18.997412 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0929 22:50:18.997558 (+ 146us) server_negotiation.cc:299] Negotiation successful
0929 22:50:18.997593 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260929 22:50:18.997658 16543 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:50:18.994999 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.147.194:44751 (local address 127.14.147.193:45367)
0929 22:50:18.995072 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0929 22:50:18.995075 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 22:50:18.995089 (+ 14us) server_negotiation.cc:407] Connection header received
0929 22:50:18.995133 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 22:50:18.995135 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 22:50:18.995148 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 22:50:18.995358 (+ 210us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 22:50:18.995562 (+ 204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:50:18.996036 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:50:18.996974 (+ 938us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:50:18.997334 (+ 360us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:50:18.997397 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 22:50:18.997407 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0929 22:50:18.997549 (+ 142us) server_negotiation.cc:299] Negotiation successful
0929 22:50:18.997589 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260929 22:50:18.997668 16276 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:50:18.994884 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.147.193:45367 (local address 127.14.147.194:44751)
0929 22:50:18.994950 (+ 66us) negotiation.cc:107] Waiting for socket to connect
0929 22:50:18.994961 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0929 22:50:18.995009 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 22:50:18.995381 (+ 372us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 22:50:18.995382 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 22:50:18.995385 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 22:50:18.995486 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 22:50:18.995488 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 22:50:18.996114 (+ 626us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 22:50:18.996115 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 22:50:18.996873 (+ 758us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 22:50:18.996875 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 22:50:18.997409 (+ 534us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 22:50:18.997410 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 22:50:18.997487 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 22:50:18.997490 (+ 3us) client_negotiation.cc:769] Sending connection context
0929 22:50:18.997535 (+ 45us) client_negotiation.cc:240] Negotiation successful
0929 22:50:18.997575 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":45}
I20260929 22:50:18.997680 16555 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:50:18.994999 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.147.195:44731 (local address 127.14.147.194:39227)
0929 22:50:18.995270 (+ 271us) negotiation.cc:107] Waiting for socket to connect
0929 22:50:18.995281 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0929 22:50:18.995343 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 22:50:18.995505 (+ 162us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 22:50:18.995506 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 22:50:18.995507 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 22:50:18.995584 (+ 77us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 22:50:18.995585 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 22:50:18.996174 (+ 589us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 22:50:18.996175 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 22:50:18.996896 (+ 721us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 22:50:18.996898 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 22:50:18.997420 (+ 522us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 22:50:18.997421 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 22:50:18.997496 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 22:50:18.997499 (+ 3us) client_negotiation.cc:769] Sending connection context
0929 22:50:18.997542 (+ 43us) client_negotiation.cc:240] Negotiation successful
0929 22:50:18.997581 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":248,"mutex_wait_us":2,"thread_start_us":185,"threads_started":1}
I20260929 22:50:18.998034 16197 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b327bea83641424e8de6b692c1ef57b9" candidate_uuid: "2bcc9256a4f84ef19cdb214bf660076f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0e77842f330e4c0c9797c75ddf7a706a" is_pre_election: true
I20260929 22:50:18.998040 16477 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b327bea83641424e8de6b692c1ef57b9" candidate_uuid: "2bcc9256a4f84ef19cdb214bf660076f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3a977c79b99544e08ef536df3e5fef57" is_pre_election: true
I20260929 22:50:18.998198 16197 raft_consensus.cc:2454] T b327bea83641424e8de6b692c1ef57b9 P 0e77842f330e4c0c9797c75ddf7a706a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2bcc9256a4f84ef19cdb214bf660076f in term 0.
I20260929 22:50:18.998198 16477 raft_consensus.cc:2454] T b327bea83641424e8de6b692c1ef57b9 P 3a977c79b99544e08ef536df3e5fef57 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2bcc9256a4f84ef19cdb214bf660076f in term 0.
I20260929 22:50:18.998426 16269 leader_election.cc:302] T b327bea83641424e8de6b692c1ef57b9 P 2bcc9256a4f84ef19cdb214bf660076f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2bcc9256a4f84ef19cdb214bf660076f, 3a977c79b99544e08ef536df3e5fef57; no voters:
I20260929 22:50:18.998549 16552 raft_consensus.cc:2781] T b327bea83641424e8de6b692c1ef57b9 P 2bcc9256a4f84ef19cdb214bf660076f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 22:50:18.998605 16552 raft_consensus.cc:482] T b327bea83641424e8de6b692c1ef57b9 P 2bcc9256a4f84ef19cdb214bf660076f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 22:50:18.998628 16552 raft_consensus.cc:3037] T b327bea83641424e8de6b692c1ef57b9 P 2bcc9256a4f84ef19cdb214bf660076f [term 0 FOLLOWER]: Advancing to term 1
I20260929 22:50:18.999594 16552 raft_consensus.cc:504] T b327bea83641424e8de6b692c1ef57b9 P 2bcc9256a4f84ef19cdb214bf660076f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e77842f330e4c0c9797c75ddf7a706a" member_type: VOTER last_known_addr { host: "127.14.147.193" port: 45367 } } peers { permanent_uuid: "2bcc9256a4f84ef19cdb214bf660076f" member_type: VOTER last_known_addr { host: "127.14.147.194" port: 46111 } } peers { permanent_uuid: "3a977c79b99544e08ef536df3e5fef57" member_type: VOTER last_known_addr { host: "127.14.147.195" port: 44731 } }
I20260929 22:50:18.999718 16552 leader_election.cc:288] T b327bea83641424e8de6b692c1ef57b9 P 2bcc9256a4f84ef19cdb214bf660076f [CANDIDATE]: Term 1 election: Requested vote from peers 0e77842f330e4c0c9797c75ddf7a706a (127.14.147.193:45367), 3a977c79b99544e08ef536df3e5fef57 (127.14.147.195:44731)
I20260929 22:50:18.999881 16477 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b327bea83641424e8de6b692c1ef57b9" candidate_uuid: "2bcc9256a4f84ef19cdb214bf660076f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3a977c79b99544e08ef536df3e5fef57"
I20260929 22:50:18.999909 16197 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b327bea83641424e8de6b692c1ef57b9" candidate_uuid: "2bcc9256a4f84ef19cdb214bf660076f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0e77842f330e4c0c9797c75ddf7a706a"
I20260929 22:50:18.999993 16197 raft_consensus.cc:3037] T b327bea83641424e8de6b692c1ef57b9 P 0e77842f330e4c0c9797c75ddf7a706a [term 0 FOLLOWER]: Advancing to term 1
I20260929 22:50:18.999989 16477 raft_consensus.cc:3037] T b327bea83641424e8de6b692c1ef57b9 P 3a977c79b99544e08ef536df3e5fef57 [term 0 FOLLOWER]: Advancing to term 1
I20260929 22:50:19.000694 16477 raft_consensus.cc:2454] T b327bea83641424e8de6b692c1ef57b9 P 3a977c79b99544e08ef536df3e5fef57 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2bcc9256a4f84ef19cdb214bf660076f in term 1.
I20260929 22:50:19.000694 16197 raft_consensus.cc:2454] T b327bea83641424e8de6b692c1ef57b9 P 0e77842f330e4c0c9797c75ddf7a706a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2bcc9256a4f84ef19cdb214bf660076f in term 1.
I20260929 22:50:19.000856 16269 leader_election.cc:302] T b327bea83641424e8de6b692c1ef57b9 P 2bcc9256a4f84ef19cdb214bf660076f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2bcc9256a4f84ef19cdb214bf660076f, 3a977c79b99544e08ef536df3e5fef57; no voters:
I20260929 22:50:19.000954 16552 raft_consensus.cc:2781] T b327bea83641424e8de6b692c1ef57b9 P 2bcc9256a4f84ef19cdb214bf660076f [term 1 FOLLOWER]: Leader election won for term 1
I20260929 22:50:19.001114 16552 raft_consensus.cc:686] T b327bea83641424e8de6b692c1ef57b9 P 2bcc9256a4f84ef19cdb214bf660076f [term 1 LEADER]: Becoming Leader. State: Replica: 2bcc9256a4f84ef19cdb214bf660076f, State: Running, Role: LEADER
I20260929 22:50:19.001206 16552 consensus_queue.cc:237] T b327bea83641424e8de6b692c1ef57b9 P 2bcc9256a4f84ef19cdb214bf660076f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e77842f330e4c0c9797c75ddf7a706a" member_type: VOTER last_known_addr { host: "127.14.147.193" port: 45367 } } peers { permanent_uuid: "2bcc9256a4f84ef19cdb214bf660076f" member_type: VOTER last_known_addr { host: "127.14.147.194" port: 46111 } } peers { permanent_uuid: "3a977c79b99544e08ef536df3e5fef57" member_type: VOTER last_known_addr { host: "127.14.147.195" port: 44731 } }
I20260929 22:50:19.002009 16039 catalog_manager.cc:5495] T b327bea83641424e8de6b692c1ef57b9 P 2bcc9256a4f84ef19cdb214bf660076f reported cstate change: term changed from 0 to 1, leader changed from <none> to 2bcc9256a4f84ef19cdb214bf660076f (127.14.147.194). New cstate: current_term: 1 leader_uuid: "2bcc9256a4f84ef19cdb214bf660076f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e77842f330e4c0c9797c75ddf7a706a" member_type: VOTER last_known_addr { host: "127.14.147.193" port: 45367 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2bcc9256a4f84ef19cdb214bf660076f" member_type: VOTER last_known_addr { host: "127.14.147.194" port: 46111 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3a977c79b99544e08ef536df3e5fef57" member_type: VOTER last_known_addr { host: "127.14.147.195" port: 44731 } health_report { overall_health: UNKNOWN } } }
I20260929 22:50:19.019979 16542 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:50:19.017441 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54296 (local address 127.14.147.194:46111)
0929 22:50:19.017487 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0929 22:50:19.017490 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 22:50:19.017533 (+ 43us) server_negotiation.cc:407] Connection header received
0929 22:50:19.017629 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 22:50:19.017630 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 22:50:19.017651 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 22:50:19.017702 (+ 51us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 22:50:19.017845 (+ 143us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:50:19.018331 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:50:19.019153 (+ 822us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:50:19.019345 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:50:19.019397 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 22:50:19.019680 (+ 283us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 22:50:19.019782 (+ 102us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 22:50:19.019849 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0929 22:50:19.019879 (+ 30us) server_negotiation.cc:299] Negotiation successful
0929 22:50:19.019901 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260929 22:50:19.021669 16197 raft_consensus.cc:1261] T b327bea83641424e8de6b692c1ef57b9 P 0e77842f330e4c0c9797c75ddf7a706a [term 1 FOLLOWER]: Refusing update from remote peer 2bcc9256a4f84ef19cdb214bf660076f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 22:50:19.021735 16477 raft_consensus.cc:1261] T b327bea83641424e8de6b692c1ef57b9 P 3a977c79b99544e08ef536df3e5fef57 [term 1 FOLLOWER]: Refusing update from remote peer 2bcc9256a4f84ef19cdb214bf660076f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 22:50:19.021970 16552 consensus_queue.cc:1035] T b327bea83641424e8de6b692c1ef57b9 P 2bcc9256a4f84ef19cdb214bf660076f [LEADER]: Connected to new peer: Peer: permanent_uuid: "0e77842f330e4c0c9797c75ddf7a706a" member_type: VOTER last_known_addr { host: "127.14.147.193" port: 45367 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 22:50:19.022109 16556 consensus_queue.cc:1035] T b327bea83641424e8de6b692c1ef57b9 P 2bcc9256a4f84ef19cdb214bf660076f [LEADER]: Connected to new peer: Peer: permanent_uuid: "3a977c79b99544e08ef536df3e5fef57" member_type: VOTER last_known_addr { host: "127.14.147.195" port: 44731 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 22:50:19.024648 16560 mvcc.cc:205] Tried to move back new op lower bound from 7334798209110245376 to 7334798209029509120. Current Snapshot: MvccSnapshot[applied={T|T < 7334798209110245376}]
I20260929 22:50:19.028961 16040 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:54240:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 22:50:19.029080 16040 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:54240:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 22:50:19.029888 16040 catalog_manager.cc:5782] T 00000000000000000000000000000000 P d20dfacee9bd43d3b147457b590bc600: Sending DeleteTablet for 3 replicas of tablet b327bea83641424e8de6b692c1ef57b9
I20260929 22:50:19.030279 16175 tablet_service.cc:1514] Processing DeleteTablet for tablet b327bea83641424e8de6b692c1ef57b9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 22:50:19 UTC) from {username='kudu', principal='kudu/127.14.147.254@KRBTEST.COM'} at 127.0.0.1:59622
I20260929 22:50:19.030294 16317 tablet_service.cc:1514] Processing DeleteTablet for tablet b327bea83641424e8de6b692c1ef57b9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 22:50:19 UTC) from {username='kudu', principal='kudu/127.14.147.254@KRBTEST.COM'} at 127.0.0.1:54288
I20260929 22:50:19.030551 16569 tablet_replica.cc:330] T b327bea83641424e8de6b692c1ef57b9 P 2bcc9256a4f84ef19cdb214bf660076f: stopping tablet replica
I20260929 22:50:19.030650 16569 raft_consensus.cc:2229] T b327bea83641424e8de6b692c1ef57b9 P 2bcc9256a4f84ef19cdb214bf660076f [term 1 LEADER]: Raft consensus shutting down.
I20260929 22:50:19.030632 16040 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:54240:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260929 22:50:19.030731 16457 tablet_service.cc:1514] Processing DeleteTablet for tablet b327bea83641424e8de6b692c1ef57b9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 22:50:19 UTC) from {username='kudu', principal='kudu/127.14.147.254@KRBTEST.COM'} at 127.0.0.1:50490
I20260929 22:50:19.030751 16569 raft_consensus.cc:2258] T b327bea83641424e8de6b692c1ef57b9 P 2bcc9256a4f84ef19cdb214bf660076f [term 1 FOLLOWER]: Raft consensus is shut down!
W20260929 22:50:19.030766 16040 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 22:50:19.030905 16570 tablet_replica.cc:330] T b327bea83641424e8de6b692c1ef57b9 P 0e77842f330e4c0c9797c75ddf7a706a: stopping tablet replica
I20260929 22:50:19.031018 16570 raft_consensus.cc:2229] T b327bea83641424e8de6b692c1ef57b9 P 0e77842f330e4c0c9797c75ddf7a706a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 22:50:19.031118 16570 raft_consensus.cc:2258] T b327bea83641424e8de6b692c1ef57b9 P 0e77842f330e4c0c9797c75ddf7a706a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 22:50:19.031598 16569 ts_tablet_manager.cc:1905] T b327bea83641424e8de6b692c1ef57b9 P 2bcc9256a4f84ef19cdb214bf660076f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 22:50:19.032018 16175 tablet_service.cc:1467] Processing CreateTablet for tablet 76fab0fef8764bdf9655355daf51c565 (DEFAULT_TABLE table=test-table [id=8ca06046ebed43dc8b4c2ddcfae5b0dd]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 22:50:19.032039 16317 tablet_service.cc:1467] Processing CreateTablet for tablet 76fab0fef8764bdf9655355daf51c565 (DEFAULT_TABLE table=test-table [id=8ca06046ebed43dc8b4c2ddcfae5b0dd]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 22:50:19.032141 16175 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 76fab0fef8764bdf9655355daf51c565. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 22:50:19.032150 16317 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 76fab0fef8764bdf9655355daf51c565. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 22:50:19.033421 16550 tablet_bootstrap.cc:492] T 76fab0fef8764bdf9655355daf51c565 P 0e77842f330e4c0c9797c75ddf7a706a: Bootstrap starting.
I20260929 22:50:19.033507 16547 tablet_bootstrap.cc:492] T 76fab0fef8764bdf9655355daf51c565 P 2bcc9256a4f84ef19cdb214bf660076f: Bootstrap starting.
I20260929 22:50:19.033725 16457 tablet_service.cc:1467] Processing CreateTablet for tablet 76fab0fef8764bdf9655355daf51c565 (DEFAULT_TABLE table=test-table [id=8ca06046ebed43dc8b4c2ddcfae5b0dd]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 22:50:19.033840 16457 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 76fab0fef8764bdf9655355daf51c565. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 22:50:19.033954 16547 tablet_bootstrap.cc:654] T 76fab0fef8764bdf9655355daf51c565 P 2bcc9256a4f84ef19cdb214bf660076f: Neither blocks nor log segments found. Creating new log.
I20260929 22:50:19.034077 16550 tablet_bootstrap.cc:654] T 76fab0fef8764bdf9655355daf51c565 P 0e77842f330e4c0c9797c75ddf7a706a: Neither blocks nor log segments found. Creating new log.
I20260929 22:50:19.034411 16569 ts_tablet_manager.cc:1918] T b327bea83641424e8de6b692c1ef57b9 P 2bcc9256a4f84ef19cdb214bf660076f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 22:50:19.034520 16569 log.cc:1196] T b327bea83641424e8de6b692c1ef57b9 P 2bcc9256a4f84ef19cdb214bf660076f: Deleting WAL directory at /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790722213368738-14927-0/minicluster-data/ts-1/wal/wals/b327bea83641424e8de6b692c1ef57b9
I20260929 22:50:19.034780 16569 ts_tablet_manager.cc:1939] T b327bea83641424e8de6b692c1ef57b9 P 2bcc9256a4f84ef19cdb214bf660076f: Deleting consensus metadata
I20260929 22:50:19.035192 16546 tablet_bootstrap.cc:492] T 76fab0fef8764bdf9655355daf51c565 P 3a977c79b99544e08ef536df3e5fef57: Bootstrap starting.
I20260929 22:50:19.035187 16025 catalog_manager.cc:4841] TS 2bcc9256a4f84ef19cdb214bf660076f (127.14.147.194:46111): tablet b327bea83641424e8de6b692c1ef57b9 (table test-table [id=7b75cc67d50343d589d7038727326749]) successfully deleted
I20260929 22:50:19.035356 16571 tablet_replica.cc:330] T b327bea83641424e8de6b692c1ef57b9 P 3a977c79b99544e08ef536df3e5fef57: stopping tablet replica
I20260929 22:50:19.035508 16571 raft_consensus.cc:2229] T b327bea83641424e8de6b692c1ef57b9 P 3a977c79b99544e08ef536df3e5fef57 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 22:50:19.035629 16546 tablet_bootstrap.cc:654] T 76fab0fef8764bdf9655355daf51c565 P 3a977c79b99544e08ef536df3e5fef57: Neither blocks nor log segments found. Creating new log.
I20260929 22:50:19.035635 16571 raft_consensus.cc:2258] T b327bea83641424e8de6b692c1ef57b9 P 3a977c79b99544e08ef536df3e5fef57 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 22:50:19.036080 16571 ts_tablet_manager.cc:1905] T b327bea83641424e8de6b692c1ef57b9 P 3a977c79b99544e08ef536df3e5fef57: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 22:50:19.036255 16546 tablet_bootstrap.cc:492] T 76fab0fef8764bdf9655355daf51c565 P 3a977c79b99544e08ef536df3e5fef57: No bootstrap required, opened a new log
I20260929 22:50:19.036309 16546 ts_tablet_manager.cc:1397] T 76fab0fef8764bdf9655355daf51c565 P 3a977c79b99544e08ef536df3e5fef57: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260929 22:50:19.036456 16546 raft_consensus.cc:348] T 76fab0fef8764bdf9655355daf51c565 P 3a977c79b99544e08ef536df3e5fef57 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e77842f330e4c0c9797c75ddf7a706a" member_type: VOTER last_known_addr { host: "127.14.147.193" port: 45367 } } peers { permanent_uuid: "2bcc9256a4f84ef19cdb214bf660076f" member_type: VOTER last_known_addr { host: "127.14.147.194" port: 46111 } } peers { permanent_uuid: "3a977c79b99544e08ef536df3e5fef57" member_type: VOTER last_known_addr { host: "127.14.147.195" port: 44731 } }
I20260929 22:50:19.036543 16546 raft_consensus.cc:374] T 76fab0fef8764bdf9655355daf51c565 P 3a977c79b99544e08ef536df3e5fef57 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 22:50:19.036563 16546 raft_consensus.cc:729] T 76fab0fef8764bdf9655355daf51c565 P 3a977c79b99544e08ef536df3e5fef57 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3a977c79b99544e08ef536df3e5fef57, State: Initialized, Role: FOLLOWER
I20260929 22:50:19.036628 16546 consensus_queue.cc:260] T 76fab0fef8764bdf9655355daf51c565 P 3a977c79b99544e08ef536df3e5fef57 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e77842f330e4c0c9797c75ddf7a706a" member_type: VOTER last_known_addr { host: "127.14.147.193" port: 45367 } } peers { permanent_uuid: "2bcc9256a4f84ef19cdb214bf660076f" member_type: VOTER last_known_addr { host: "127.14.147.194" port: 46111 } } peers { permanent_uuid: "3a977c79b99544e08ef536df3e5fef57" member_type: VOTER last_known_addr { host: "127.14.147.195" port: 44731 } }
I20260929 22:50:19.036717 16546 ts_tablet_manager.cc:1428] T 76fab0fef8764bdf9655355daf51c565 P 3a977c79b99544e08ef536df3e5fef57: Time spent starting tablet: real 0.000s user 0.000s sys 0.001s
I20260929 22:50:19.037484 16571 ts_tablet_manager.cc:1918] T b327bea83641424e8de6b692c1ef57b9 P 3a977c79b99544e08ef536df3e5fef57: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 22:50:19.037546 16571 log.cc:1196] T b327bea83641424e8de6b692c1ef57b9 P 3a977c79b99544e08ef536df3e5fef57: Deleting WAL directory at /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790722213368738-14927-0/minicluster-data/ts-2/wal/wals/b327bea83641424e8de6b692c1ef57b9
I20260929 22:50:19.037568 16570 ts_tablet_manager.cc:1905] T b327bea83641424e8de6b692c1ef57b9 P 0e77842f330e4c0c9797c75ddf7a706a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 22:50:19.037886 16571 ts_tablet_manager.cc:1939] T b327bea83641424e8de6b692c1ef57b9 P 3a977c79b99544e08ef536df3e5fef57: Deleting consensus metadata
I20260929 22:50:19.038291 16025 catalog_manager.cc:4841] TS 3a977c79b99544e08ef536df3e5fef57 (127.14.147.195:44731): tablet b327bea83641424e8de6b692c1ef57b9 (table test-table [id=7b75cc67d50343d589d7038727326749]) successfully deleted
I20260929 22:50:19.039332 16570 ts_tablet_manager.cc:1918] T b327bea83641424e8de6b692c1ef57b9 P 0e77842f330e4c0c9797c75ddf7a706a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 22:50:19.039410 16570 log.cc:1196] T b327bea83641424e8de6b692c1ef57b9 P 0e77842f330e4c0c9797c75ddf7a706a: Deleting WAL directory at /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790722213368738-14927-0/minicluster-data/ts-0/wal/wals/b327bea83641424e8de6b692c1ef57b9
I20260929 22:50:19.039678 16570 ts_tablet_manager.cc:1939] T b327bea83641424e8de6b692c1ef57b9 P 0e77842f330e4c0c9797c75ddf7a706a: Deleting consensus metadata
I20260929 22:50:19.040071 16026 catalog_manager.cc:4841] TS 0e77842f330e4c0c9797c75ddf7a706a (127.14.147.193:45367): tablet b327bea83641424e8de6b692c1ef57b9 (table test-table [id=7b75cc67d50343d589d7038727326749]) successfully deleted
I20260929 22:50:19.043834 16547 tablet_bootstrap.cc:492] T 76fab0fef8764bdf9655355daf51c565 P 2bcc9256a4f84ef19cdb214bf660076f: No bootstrap required, opened a new log
I20260929 22:50:19.043898 16547 ts_tablet_manager.cc:1397] T 76fab0fef8764bdf9655355daf51c565 P 2bcc9256a4f84ef19cdb214bf660076f: Time spent bootstrapping tablet: real 0.010s user 0.001s sys 0.000s
I20260929 22:50:19.044057 16547 raft_consensus.cc:348] T 76fab0fef8764bdf9655355daf51c565 P 2bcc9256a4f84ef19cdb214bf660076f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e77842f330e4c0c9797c75ddf7a706a" member_type: VOTER last_known_addr { host: "127.14.147.193" port: 45367 } } peers { permanent_uuid: "2bcc9256a4f84ef19cdb214bf660076f" member_type: VOTER last_known_addr { host: "127.14.147.194" port: 46111 } } peers { permanent_uuid: "3a977c79b99544e08ef536df3e5fef57" member_type: VOTER last_known_addr { host: "127.14.147.195" port: 44731 } }
I20260929 22:50:19.044124 16547 raft_consensus.cc:374] T 76fab0fef8764bdf9655355daf51c565 P 2bcc9256a4f84ef19cdb214bf660076f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 22:50:19.044142 16547 raft_consensus.cc:729] T 76fab0fef8764bdf9655355daf51c565 P 2bcc9256a4f84ef19cdb214bf660076f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2bcc9256a4f84ef19cdb214bf660076f, State: Initialized, Role: FOLLOWER
I20260929 22:50:19.044196 16547 consensus_queue.cc:260] T 76fab0fef8764bdf9655355daf51c565 P 2bcc9256a4f84ef19cdb214bf660076f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e77842f330e4c0c9797c75ddf7a706a" member_type: VOTER last_known_addr { host: "127.14.147.193" port: 45367 } } peers { permanent_uuid: "2bcc9256a4f84ef19cdb214bf660076f" member_type: VOTER last_known_addr { host: "127.14.147.194" port: 46111 } } peers { permanent_uuid: "3a977c79b99544e08ef536df3e5fef57" member_type: VOTER last_known_addr { host: "127.14.147.195" port: 44731 } }
I20260929 22:50:19.044294 16547 ts_tablet_manager.cc:1428] T 76fab0fef8764bdf9655355daf51c565 P 2bcc9256a4f84ef19cdb214bf660076f: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260929 22:50:19.044499 16550 tablet_bootstrap.cc:492] T 76fab0fef8764bdf9655355daf51c565 P 0e77842f330e4c0c9797c75ddf7a706a: No bootstrap required, opened a new log
I20260929 22:50:19.044567 16550 ts_tablet_manager.cc:1397] T 76fab0fef8764bdf9655355daf51c565 P 0e77842f330e4c0c9797c75ddf7a706a: Time spent bootstrapping tablet: real 0.011s user 0.001s sys 0.000s
I20260929 22:50:19.044730 16550 raft_consensus.cc:348] T 76fab0fef8764bdf9655355daf51c565 P 0e77842f330e4c0c9797c75ddf7a706a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e77842f330e4c0c9797c75ddf7a706a" member_type: VOTER last_known_addr { host: "127.14.147.193" port: 45367 } } peers { permanent_uuid: "2bcc9256a4f84ef19cdb214bf660076f" member_type: VOTER last_known_addr { host: "127.14.147.194" port: 46111 } } peers { permanent_uuid: "3a977c79b99544e08ef536df3e5fef57" member_type: VOTER last_known_addr { host: "127.14.147.195" port: 44731 } }
I20260929 22:50:19.044806 16550 raft_consensus.cc:374] T 76fab0fef8764bdf9655355daf51c565 P 0e77842f330e4c0c9797c75ddf7a706a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 22:50:19.044831 16550 raft_consensus.cc:729] T 76fab0fef8764bdf9655355daf51c565 P 0e77842f330e4c0c9797c75ddf7a706a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0e77842f330e4c0c9797c75ddf7a706a, State: Initialized, Role: FOLLOWER
I20260929 22:50:19.044871 16550 consensus_queue.cc:260] T 76fab0fef8764bdf9655355daf51c565 P 0e77842f330e4c0c9797c75ddf7a706a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e77842f330e4c0c9797c75ddf7a706a" member_type: VOTER last_known_addr { host: "127.14.147.193" port: 45367 } } peers { permanent_uuid: "2bcc9256a4f84ef19cdb214bf660076f" member_type: VOTER last_known_addr { host: "127.14.147.194" port: 46111 } } peers { permanent_uuid: "3a977c79b99544e08ef536df3e5fef57" member_type: VOTER last_known_addr { host: "127.14.147.195" port: 44731 } }
I20260929 22:50:19.044962 16550 ts_tablet_manager.cc:1428] T 76fab0fef8764bdf9655355daf51c565 P 0e77842f330e4c0c9797c75ddf7a706a: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
W20260929 22:50:19.053750 16244 tablet.cc:2369] T 76fab0fef8764bdf9655355daf51c565 P 0e77842f330e4c0c9797c75ddf7a706a: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260929 22:50:19.107395 16524 tablet.cc:2369] T 76fab0fef8764bdf9655355daf51c565 P 3a977c79b99544e08ef536df3e5fef57: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 22:50:19.248889 16566 raft_consensus.cc:482] T 76fab0fef8764bdf9655355daf51c565 P 2bcc9256a4f84ef19cdb214bf660076f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 22:50:19.249009 16566 raft_consensus.cc:504] T 76fab0fef8764bdf9655355daf51c565 P 2bcc9256a4f84ef19cdb214bf660076f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e77842f330e4c0c9797c75ddf7a706a" member_type: VOTER last_known_addr { host: "127.14.147.193" port: 45367 } } peers { permanent_uuid: "2bcc9256a4f84ef19cdb214bf660076f" member_type: VOTER last_known_addr { host: "127.14.147.194" port: 46111 } } peers { permanent_uuid: "3a977c79b99544e08ef536df3e5fef57" member_type: VOTER last_known_addr { host: "127.14.147.195" port: 44731 } }
I20260929 22:50:19.249195 16566 leader_election.cc:288] T 76fab0fef8764bdf9655355daf51c565 P 2bcc9256a4f84ef19cdb214bf660076f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0e77842f330e4c0c9797c75ddf7a706a (127.14.147.193:45367), 3a977c79b99544e08ef536df3e5fef57 (127.14.147.195:44731)
I20260929 22:50:19.249436 16477 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "76fab0fef8764bdf9655355daf51c565" candidate_uuid: "2bcc9256a4f84ef19cdb214bf660076f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3a977c79b99544e08ef536df3e5fef57" is_pre_election: true
I20260929 22:50:19.249436 16197 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "76fab0fef8764bdf9655355daf51c565" candidate_uuid: "2bcc9256a4f84ef19cdb214bf660076f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0e77842f330e4c0c9797c75ddf7a706a" is_pre_election: true
I20260929 22:50:19.249567 16197 raft_consensus.cc:2454] T 76fab0fef8764bdf9655355daf51c565 P 0e77842f330e4c0c9797c75ddf7a706a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2bcc9256a4f84ef19cdb214bf660076f in term 0.
I20260929 22:50:19.249567 16477 raft_consensus.cc:2454] T 76fab0fef8764bdf9655355daf51c565 P 3a977c79b99544e08ef536df3e5fef57 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2bcc9256a4f84ef19cdb214bf660076f in term 0.
I20260929 22:50:19.249759 16269 leader_election.cc:302] T 76fab0fef8764bdf9655355daf51c565 P 2bcc9256a4f84ef19cdb214bf660076f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2bcc9256a4f84ef19cdb214bf660076f, 3a977c79b99544e08ef536df3e5fef57; no voters:
I20260929 22:50:19.249873 16566 raft_consensus.cc:2781] T 76fab0fef8764bdf9655355daf51c565 P 2bcc9256a4f84ef19cdb214bf660076f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 22:50:19.249929 16566 raft_consensus.cc:482] T 76fab0fef8764bdf9655355daf51c565 P 2bcc9256a4f84ef19cdb214bf660076f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 22:50:19.249944 16566 raft_consensus.cc:3037] T 76fab0fef8764bdf9655355daf51c565 P 2bcc9256a4f84ef19cdb214bf660076f [term 0 FOLLOWER]: Advancing to term 1
I20260929 22:50:19.250674 16566 raft_consensus.cc:504] T 76fab0fef8764bdf9655355daf51c565 P 2bcc9256a4f84ef19cdb214bf660076f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e77842f330e4c0c9797c75ddf7a706a" member_type: VOTER last_known_addr { host: "127.14.147.193" port: 45367 } } peers { permanent_uuid: "2bcc9256a4f84ef19cdb214bf660076f" member_type: VOTER last_known_addr { host: "127.14.147.194" port: 46111 } } peers { permanent_uuid: "3a977c79b99544e08ef536df3e5fef57" member_type: VOTER last_known_addr { host: "127.14.147.195" port: 44731 } }
I20260929 22:50:19.250793 16566 leader_election.cc:288] T 76fab0fef8764bdf9655355daf51c565 P 2bcc9256a4f84ef19cdb214bf660076f [CANDIDATE]: Term 1 election: Requested vote from peers 0e77842f330e4c0c9797c75ddf7a706a (127.14.147.193:45367), 3a977c79b99544e08ef536df3e5fef57 (127.14.147.195:44731)
I20260929 22:50:19.251189 16197 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "76fab0fef8764bdf9655355daf51c565" candidate_uuid: "2bcc9256a4f84ef19cdb214bf660076f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0e77842f330e4c0c9797c75ddf7a706a"
I20260929 22:50:19.251183 16477 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "76fab0fef8764bdf9655355daf51c565" candidate_uuid: "2bcc9256a4f84ef19cdb214bf660076f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3a977c79b99544e08ef536df3e5fef57"
I20260929 22:50:19.251274 16477 raft_consensus.cc:3037] T 76fab0fef8764bdf9655355daf51c565 P 3a977c79b99544e08ef536df3e5fef57 [term 0 FOLLOWER]: Advancing to term 1
I20260929 22:50:19.251274 16197 raft_consensus.cc:3037] T 76fab0fef8764bdf9655355daf51c565 P 0e77842f330e4c0c9797c75ddf7a706a [term 0 FOLLOWER]: Advancing to term 1
I20260929 22:50:19.252264 16477 raft_consensus.cc:2454] T 76fab0fef8764bdf9655355daf51c565 P 3a977c79b99544e08ef536df3e5fef57 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2bcc9256a4f84ef19cdb214bf660076f in term 1.
I20260929 22:50:19.252264 16197 raft_consensus.cc:2454] T 76fab0fef8764bdf9655355daf51c565 P 0e77842f330e4c0c9797c75ddf7a706a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2bcc9256a4f84ef19cdb214bf660076f in term 1.
I20260929 22:50:19.252468 16269 leader_election.cc:302] T 76fab0fef8764bdf9655355daf51c565 P 2bcc9256a4f84ef19cdb214bf660076f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2bcc9256a4f84ef19cdb214bf660076f, 3a977c79b99544e08ef536df3e5fef57; no voters:
I20260929 22:50:19.252666 16566 raft_consensus.cc:2781] T 76fab0fef8764bdf9655355daf51c565 P 2bcc9256a4f84ef19cdb214bf660076f [term 1 FOLLOWER]: Leader election won for term 1
I20260929 22:50:19.252750 16566 raft_consensus.cc:686] T 76fab0fef8764bdf9655355daf51c565 P 2bcc9256a4f84ef19cdb214bf660076f [term 1 LEADER]: Becoming Leader. State: Replica: 2bcc9256a4f84ef19cdb214bf660076f, State: Running, Role: LEADER
I20260929 22:50:19.252799 16566 consensus_queue.cc:237] T 76fab0fef8764bdf9655355daf51c565 P 2bcc9256a4f84ef19cdb214bf660076f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e77842f330e4c0c9797c75ddf7a706a" member_type: VOTER last_known_addr { host: "127.14.147.193" port: 45367 } } peers { permanent_uuid: "2bcc9256a4f84ef19cdb214bf660076f" member_type: VOTER last_known_addr { host: "127.14.147.194" port: 46111 } } peers { permanent_uuid: "3a977c79b99544e08ef536df3e5fef57" member_type: VOTER last_known_addr { host: "127.14.147.195" port: 44731 } }
I20260929 22:50:19.253545 16040 catalog_manager.cc:5495] T 76fab0fef8764bdf9655355daf51c565 P 2bcc9256a4f84ef19cdb214bf660076f reported cstate change: term changed from 0 to 1, leader changed from <none> to 2bcc9256a4f84ef19cdb214bf660076f (127.14.147.194). New cstate: current_term: 1 leader_uuid: "2bcc9256a4f84ef19cdb214bf660076f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e77842f330e4c0c9797c75ddf7a706a" member_type: VOTER last_known_addr { host: "127.14.147.193" port: 45367 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2bcc9256a4f84ef19cdb214bf660076f" member_type: VOTER last_known_addr { host: "127.14.147.194" port: 46111 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3a977c79b99544e08ef536df3e5fef57" member_type: VOTER last_known_addr { host: "127.14.147.195" port: 44731 } health_report { overall_health: UNKNOWN } } }
Sep 29 22:50:19 dist-test-slave-98mk krb5kdc[15995](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790722218, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.147.254@KRBTEST.COM for kudu/127.14.147.254@KRBTEST.COM
I20260929 22:50:19.298724 16093 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:50:19.292277 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54256 (local address 127.14.147.254:46225)
0929 22:50:19.292345 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0929 22:50:19.292347 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 22:50:19.292433 (+ 86us) server_negotiation.cc:407] Connection header received
0929 22:50:19.292482 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 22:50:19.292483 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 22:50:19.292491 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 22:50:19.292544 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0929 22:50:19.292951 (+ 407us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:50:19.293472 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:50:19.294290 (+ 818us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:50:19.294476 (+ 186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:50:19.294548 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 22:50:19.296542 (+ 1994us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 22:50:19.296543 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 22:50:19.296544 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 22:50:19.296550 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0929 22:50:19.297851 (+ 1301us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 22:50:19.298179 (+ 328us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 22:50:19.298180 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 22:50:19.298180 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 22:50:19.298226 (+ 46us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 22:50:19.298346 (+ 120us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 22:50:19.298346 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 22:50:19.298346 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 22:50:19.298439 (+ 93us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 22:50:19.298487 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0929 22:50:19.298590 (+ 103us) server_negotiation.cc:299] Negotiation successful
0929 22:50:19.298658 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260929 22:50:19.298816 16581 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:50:19.292190 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.147.254:46225 (local address 127.0.0.1:54256)
0929 22:50:19.292367 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0929 22:50:19.292377 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0929 22:50:19.292442 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 22:50:19.292556 (+ 114us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 22:50:19.292557 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 22:50:19.292837 (+ 280us) client_negotiation.cc:189] Negotiated authn=SASL
0929 22:50:19.292888 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 22:50:19.292889 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 22:50:19.293581 (+ 692us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 22:50:19.293582 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 22:50:19.294205 (+ 623us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 22:50:19.294207 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 22:50:19.294547 (+ 340us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 22:50:19.294547 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 22:50:19.294588 (+ 41us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 22:50:19.295072 (+ 484us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 22:50:19.295076 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0929 22:50:19.296452 (+ 1376us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 22:50:19.297922 (+ 1470us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 22:50:19.297929 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 22:50:19.297931 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 22:50:19.298126 (+ 195us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 22:50:19.298268 (+ 142us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 22:50:19.298268 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 22:50:19.298268 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 22:50:19.298314 (+ 46us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 22:50:19.298484 (+ 170us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 22:50:19.298488 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 22:50:19.298531 (+ 43us) client_negotiation.cc:769] Sending connection context
0929 22:50:19.298575 (+ 44us) client_negotiation.cc:240] Negotiation successful
0929 22:50:19.298713 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":148,"mutex_wait_us":53,"thread_start_us":77,"threads_started":1}
I20260929 22:50:19.300683 16040 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.14.147.254@KRBTEST.COM'} at 127.0.0.1:54256:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260929 22:50:19.300861 16040 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 22:50:19.302115 16317 tablet_service.cc:1467] Processing CreateTablet for tablet 409d925c4ae143a5be051496c7d2f153 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=b1c235204fdf4c82b194783e70474b4f]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260929 22:50:19.302109 16175 tablet_service.cc:1467] Processing CreateTablet for tablet 409d925c4ae143a5be051496c7d2f153 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=b1c235204fdf4c82b194783e70474b4f]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260929 22:50:19.302138 16457 tablet_service.cc:1467] Processing CreateTablet for tablet 409d925c4ae143a5be051496c7d2f153 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=b1c235204fdf4c82b194783e70474b4f]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260929 22:50:19.302275 16457 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 409d925c4ae143a5be051496c7d2f153. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 22:50:19.302276 16175 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 409d925c4ae143a5be051496c7d2f153. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 22:50:19.302275 16317 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 409d925c4ae143a5be051496c7d2f153. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 22:50:19.303462 16546 tablet_bootstrap.cc:492] T 409d925c4ae143a5be051496c7d2f153 P 3a977c79b99544e08ef536df3e5fef57: Bootstrap starting.
I20260929 22:50:19.303771 16547 tablet_bootstrap.cc:492] T 409d925c4ae143a5be051496c7d2f153 P 2bcc9256a4f84ef19cdb214bf660076f: Bootstrap starting.
I20260929 22:50:19.303870 16546 tablet_bootstrap.cc:654] T 409d925c4ae143a5be051496c7d2f153 P 3a977c79b99544e08ef536df3e5fef57: Neither blocks nor log segments found. Creating new log.
I20260929 22:50:19.304147 16547 tablet_bootstrap.cc:654] T 409d925c4ae143a5be051496c7d2f153 P 2bcc9256a4f84ef19cdb214bf660076f: Neither blocks nor log segments found. Creating new log.
I20260929 22:50:19.304381 16550 tablet_bootstrap.cc:492] T 409d925c4ae143a5be051496c7d2f153 P 0e77842f330e4c0c9797c75ddf7a706a: Bootstrap starting.
I20260929 22:50:19.304613 16546 tablet_bootstrap.cc:492] T 409d925c4ae143a5be051496c7d2f153 P 3a977c79b99544e08ef536df3e5fef57: No bootstrap required, opened a new log
I20260929 22:50:19.304664 16546 ts_tablet_manager.cc:1397] T 409d925c4ae143a5be051496c7d2f153 P 3a977c79b99544e08ef536df3e5fef57: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260929 22:50:19.304769 16550 tablet_bootstrap.cc:654] T 409d925c4ae143a5be051496c7d2f153 P 0e77842f330e4c0c9797c75ddf7a706a: Neither blocks nor log segments found. Creating new log.
I20260929 22:50:19.304807 16546 raft_consensus.cc:348] T 409d925c4ae143a5be051496c7d2f153 P 3a977c79b99544e08ef536df3e5fef57 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e77842f330e4c0c9797c75ddf7a706a" member_type: VOTER last_known_addr { host: "127.14.147.193" port: 45367 } } peers { permanent_uuid: "2bcc9256a4f84ef19cdb214bf660076f" member_type: VOTER last_known_addr { host: "127.14.147.194" port: 46111 } } peers { permanent_uuid: "3a977c79b99544e08ef536df3e5fef57" member_type: VOTER last_known_addr { host: "127.14.147.195" port: 44731 } }
I20260929 22:50:19.304888 16546 raft_consensus.cc:374] T 409d925c4ae143a5be051496c7d2f153 P 3a977c79b99544e08ef536df3e5fef57 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 22:50:19.304911 16546 raft_consensus.cc:729] T 409d925c4ae143a5be051496c7d2f153 P 3a977c79b99544e08ef536df3e5fef57 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3a977c79b99544e08ef536df3e5fef57, State: Initialized, Role: FOLLOWER
I20260929 22:50:19.304962 16546 consensus_queue.cc:260] T 409d925c4ae143a5be051496c7d2f153 P 3a977c79b99544e08ef536df3e5fef57 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e77842f330e4c0c9797c75ddf7a706a" member_type: VOTER last_known_addr { host: "127.14.147.193" port: 45367 } } peers { permanent_uuid: "2bcc9256a4f84ef19cdb214bf660076f" member_type: VOTER last_known_addr { host: "127.14.147.194" port: 46111 } } peers { permanent_uuid: "3a977c79b99544e08ef536df3e5fef57" member_type: VOTER last_known_addr { host: "127.14.147.195" port: 44731 } }
I20260929 22:50:19.305070 16546 ts_tablet_manager.cc:1428] T 409d925c4ae143a5be051496c7d2f153 P 3a977c79b99544e08ef536df3e5fef57: Time spent starting tablet: real 0.000s user 0.000s sys 0.001s
I20260929 22:50:19.305184 16547 tablet_bootstrap.cc:492] T 409d925c4ae143a5be051496c7d2f153 P 2bcc9256a4f84ef19cdb214bf660076f: No bootstrap required, opened a new log
I20260929 22:50:19.305150 16553 tablet_replica.cc:439] T 409d925c4ae143a5be051496c7d2f153 P 3a977c79b99544e08ef536df3e5fef57: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e77842f330e4c0c9797c75ddf7a706a" member_type: VOTER last_known_addr { host: "127.14.147.193" port: 45367 } } peers { permanent_uuid: "2bcc9256a4f84ef19cdb214bf660076f" member_type: VOTER last_known_addr { host: "127.14.147.194" port: 46111 } } peers { permanent_uuid: "3a977c79b99544e08ef536df3e5fef57" member_type: VOTER last_known_addr { host: "127.14.147.195" port: 44731 } } }
I20260929 22:50:19.305233 16547 ts_tablet_manager.cc:1397] T 409d925c4ae143a5be051496c7d2f153 P 2bcc9256a4f84ef19cdb214bf660076f: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260929 22:50:19.305243 16553 tablet_replica.cc:442] T 409d925c4ae143a5be051496c7d2f153 P 3a977c79b99544e08ef536df3e5fef57: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 22:50:19.305341 16547 raft_consensus.cc:348] T 409d925c4ae143a5be051496c7d2f153 P 2bcc9256a4f84ef19cdb214bf660076f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e77842f330e4c0c9797c75ddf7a706a" member_type: VOTER last_known_addr { host: "127.14.147.193" port: 45367 } } peers { permanent_uuid: "2bcc9256a4f84ef19cdb214bf660076f" member_type: VOTER last_known_addr { host: "127.14.147.194" port: 46111 } } peers { permanent_uuid: "3a977c79b99544e08ef536df3e5fef57" member_type: VOTER last_known_addr { host: "127.14.147.195" port: 44731 } }
I20260929 22:50:19.305410 16547 raft_consensus.cc:374] T 409d925c4ae143a5be051496c7d2f153 P 2bcc9256a4f84ef19cdb214bf660076f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 22:50:19.305436 16547 raft_consensus.cc:729] T 409d925c4ae143a5be051496c7d2f153 P 2bcc9256a4f84ef19cdb214bf660076f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2bcc9256a4f84ef19cdb214bf660076f, State: Initialized, Role: FOLLOWER
I20260929 22:50:19.305483 16547 consensus_queue.cc:260] T 409d925c4ae143a5be051496c7d2f153 P 2bcc9256a4f84ef19cdb214bf660076f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e77842f330e4c0c9797c75ddf7a706a" member_type: VOTER last_known_addr { host: "127.14.147.193" port: 45367 } } peers { permanent_uuid: "2bcc9256a4f84ef19cdb214bf660076f" member_type: VOTER last_known_addr { host: "127.14.147.194" port: 46111 } } peers { permanent_uuid: "3a977c79b99544e08ef536df3e5fef57" member_type: VOTER last_known_addr { host: "127.14.147.195" port: 44731 } }
I20260929 22:50:19.305593 16547 ts_tablet_manager.cc:1428] T 409d925c4ae143a5be051496c7d2f153 P 2bcc9256a4f84ef19cdb214bf660076f: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260929 22:50:19.305660 16550 tablet_bootstrap.cc:492] T 409d925c4ae143a5be051496c7d2f153 P 0e77842f330e4c0c9797c75ddf7a706a: No bootstrap required, opened a new log
I20260929 22:50:19.305622 16566 tablet_replica.cc:439] T 409d925c4ae143a5be051496c7d2f153 P 2bcc9256a4f84ef19cdb214bf660076f: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e77842f330e4c0c9797c75ddf7a706a" member_type: VOTER last_known_addr { host: "127.14.147.193" port: 45367 } } peers { permanent_uuid: "2bcc9256a4f84ef19cdb214bf660076f" member_type: VOTER last_known_addr { host: "127.14.147.194" port: 46111 } } peers { permanent_uuid: "3a977c79b99544e08ef536df3e5fef57" member_type: VOTER last_known_addr { host: "127.14.147.195" port: 44731 } } }
I20260929 22:50:19.305701 16566 tablet_replica.cc:442] T 409d925c4ae143a5be051496c7d2f153 P 2bcc9256a4f84ef19cdb214bf660076f: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 22:50:19.305707 16550 ts_tablet_manager.cc:1397] T 409d925c4ae143a5be051496c7d2f153 P 0e77842f330e4c0c9797c75ddf7a706a: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260929 22:50:19.305855 16550 raft_consensus.cc:348] T 409d925c4ae143a5be051496c7d2f153 P 0e77842f330e4c0c9797c75ddf7a706a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e77842f330e4c0c9797c75ddf7a706a" member_type: VOTER last_known_addr { host: "127.14.147.193" port: 45367 } } peers { permanent_uuid: "2bcc9256a4f84ef19cdb214bf660076f" member_type: VOTER last_known_addr { host: "127.14.147.194" port: 46111 } } peers { permanent_uuid: "3a977c79b99544e08ef536df3e5fef57" member_type: VOTER last_known_addr { host: "127.14.147.195" port: 44731 } }
I20260929 22:50:19.305923 16550 raft_consensus.cc:374] T 409d925c4ae143a5be051496c7d2f153 P 0e77842f330e4c0c9797c75ddf7a706a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 22:50:19.305949 16550 raft_consensus.cc:729] T 409d925c4ae143a5be051496c7d2f153 P 0e77842f330e4c0c9797c75ddf7a706a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0e77842f330e4c0c9797c75ddf7a706a, State: Initialized, Role: FOLLOWER
I20260929 22:50:19.305987 16550 consensus_queue.cc:260] T 409d925c4ae143a5be051496c7d2f153 P 0e77842f330e4c0c9797c75ddf7a706a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e77842f330e4c0c9797c75ddf7a706a" member_type: VOTER last_known_addr { host: "127.14.147.193" port: 45367 } } peers { permanent_uuid: "2bcc9256a4f84ef19cdb214bf660076f" member_type: VOTER last_known_addr { host: "127.14.147.194" port: 46111 } } peers { permanent_uuid: "3a977c79b99544e08ef536df3e5fef57" member_type: VOTER last_known_addr { host: "127.14.147.195" port: 44731 } }
I20260929 22:50:19.306071 16550 ts_tablet_manager.cc:1428] T 409d925c4ae143a5be051496c7d2f153 P 0e77842f330e4c0c9797c75ddf7a706a: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260929 22:50:19.306121 16554 tablet_replica.cc:439] T 409d925c4ae143a5be051496c7d2f153 P 0e77842f330e4c0c9797c75ddf7a706a: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e77842f330e4c0c9797c75ddf7a706a" member_type: VOTER last_known_addr { host: "127.14.147.193" port: 45367 } } peers { permanent_uuid: "2bcc9256a4f84ef19cdb214bf660076f" member_type: VOTER last_known_addr { host: "127.14.147.194" port: 46111 } } peers { permanent_uuid: "3a977c79b99544e08ef536df3e5fef57" member_type: VOTER last_known_addr { host: "127.14.147.195" port: 44731 } } }
I20260929 22:50:19.306196 16554 tablet_replica.cc:442] T 409d925c4ae143a5be051496c7d2f153 P 0e77842f330e4c0c9797c75ddf7a706a: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 22:50:19.396749 16566 raft_consensus.cc:482] T 409d925c4ae143a5be051496c7d2f153 P 2bcc9256a4f84ef19cdb214bf660076f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 22:50:19.396854 16566 raft_consensus.cc:504] T 409d925c4ae143a5be051496c7d2f153 P 2bcc9256a4f84ef19cdb214bf660076f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e77842f330e4c0c9797c75ddf7a706a" member_type: VOTER last_known_addr { host: "127.14.147.193" port: 45367 } } peers { permanent_uuid: "2bcc9256a4f84ef19cdb214bf660076f" member_type: VOTER last_known_addr { host: "127.14.147.194" port: 46111 } } peers { permanent_uuid: "3a977c79b99544e08ef536df3e5fef57" member_type: VOTER last_known_addr { host: "127.14.147.195" port: 44731 } }
I20260929 22:50:19.397023 16566 leader_election.cc:288] T 409d925c4ae143a5be051496c7d2f153 P 2bcc9256a4f84ef19cdb214bf660076f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0e77842f330e4c0c9797c75ddf7a706a (127.14.147.193:45367), 3a977c79b99544e08ef536df3e5fef57 (127.14.147.195:44731)
I20260929 22:50:19.397202 16197 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "409d925c4ae143a5be051496c7d2f153" candidate_uuid: "2bcc9256a4f84ef19cdb214bf660076f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0e77842f330e4c0c9797c75ddf7a706a" is_pre_election: true
I20260929 22:50:19.397208 16477 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "409d925c4ae143a5be051496c7d2f153" candidate_uuid: "2bcc9256a4f84ef19cdb214bf660076f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3a977c79b99544e08ef536df3e5fef57" is_pre_election: true
I20260929 22:50:19.397297 16197 raft_consensus.cc:2454] T 409d925c4ae143a5be051496c7d2f153 P 0e77842f330e4c0c9797c75ddf7a706a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2bcc9256a4f84ef19cdb214bf660076f in term 0.
I20260929 22:50:19.397297 16477 raft_consensus.cc:2454] T 409d925c4ae143a5be051496c7d2f153 P 3a977c79b99544e08ef536df3e5fef57 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2bcc9256a4f84ef19cdb214bf660076f in term 0.
I20260929 22:50:19.397446 16270 leader_election.cc:302] T 409d925c4ae143a5be051496c7d2f153 P 2bcc9256a4f84ef19cdb214bf660076f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0e77842f330e4c0c9797c75ddf7a706a, 2bcc9256a4f84ef19cdb214bf660076f; no voters:
I20260929 22:50:19.397542 16566 raft_consensus.cc:2781] T 409d925c4ae143a5be051496c7d2f153 P 2bcc9256a4f84ef19cdb214bf660076f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 22:50:19.397574 16566 raft_consensus.cc:482] T 409d925c4ae143a5be051496c7d2f153 P 2bcc9256a4f84ef19cdb214bf660076f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 22:50:19.397599 16566 raft_consensus.cc:3037] T 409d925c4ae143a5be051496c7d2f153 P 2bcc9256a4f84ef19cdb214bf660076f [term 0 FOLLOWER]: Advancing to term 1
I20260929 22:50:19.398268 16566 raft_consensus.cc:504] T 409d925c4ae143a5be051496c7d2f153 P 2bcc9256a4f84ef19cdb214bf660076f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e77842f330e4c0c9797c75ddf7a706a" member_type: VOTER last_known_addr { host: "127.14.147.193" port: 45367 } } peers { permanent_uuid: "2bcc9256a4f84ef19cdb214bf660076f" member_type: VOTER last_known_addr { host: "127.14.147.194" port: 46111 } } peers { permanent_uuid: "3a977c79b99544e08ef536df3e5fef57" member_type: VOTER last_known_addr { host: "127.14.147.195" port: 44731 } }
I20260929 22:50:19.398375 16566 leader_election.cc:288] T 409d925c4ae143a5be051496c7d2f153 P 2bcc9256a4f84ef19cdb214bf660076f [CANDIDATE]: Term 1 election: Requested vote from peers 0e77842f330e4c0c9797c75ddf7a706a (127.14.147.193:45367), 3a977c79b99544e08ef536df3e5fef57 (127.14.147.195:44731)
I20260929 22:50:19.398500 16197 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "409d925c4ae143a5be051496c7d2f153" candidate_uuid: "2bcc9256a4f84ef19cdb214bf660076f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0e77842f330e4c0c9797c75ddf7a706a"
I20260929 22:50:19.398576 16197 raft_consensus.cc:3037] T 409d925c4ae143a5be051496c7d2f153 P 0e77842f330e4c0c9797c75ddf7a706a [term 0 FOLLOWER]: Advancing to term 1
I20260929 22:50:19.398546 16477 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "409d925c4ae143a5be051496c7d2f153" candidate_uuid: "2bcc9256a4f84ef19cdb214bf660076f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3a977c79b99544e08ef536df3e5fef57"
I20260929 22:50:19.398610 16477 raft_consensus.cc:3037] T 409d925c4ae143a5be051496c7d2f153 P 3a977c79b99544e08ef536df3e5fef57 [term 0 FOLLOWER]: Advancing to term 1
I20260929 22:50:19.399277 16477 raft_consensus.cc:2454] T 409d925c4ae143a5be051496c7d2f153 P 3a977c79b99544e08ef536df3e5fef57 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2bcc9256a4f84ef19cdb214bf660076f in term 1.
I20260929 22:50:19.399277 16197 raft_consensus.cc:2454] T 409d925c4ae143a5be051496c7d2f153 P 0e77842f330e4c0c9797c75ddf7a706a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2bcc9256a4f84ef19cdb214bf660076f in term 1.
I20260929 22:50:19.399468 16270 leader_election.cc:302] T 409d925c4ae143a5be051496c7d2f153 P 2bcc9256a4f84ef19cdb214bf660076f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0e77842f330e4c0c9797c75ddf7a706a, 2bcc9256a4f84ef19cdb214bf660076f; no voters:
I20260929 22:50:19.399555 16566 raft_consensus.cc:2781] T 409d925c4ae143a5be051496c7d2f153 P 2bcc9256a4f84ef19cdb214bf660076f [term 1 FOLLOWER]: Leader election won for term 1
I20260929 22:50:19.399605 16566 raft_consensus.cc:686] T 409d925c4ae143a5be051496c7d2f153 P 2bcc9256a4f84ef19cdb214bf660076f [term 1 LEADER]: Becoming Leader. State: Replica: 2bcc9256a4f84ef19cdb214bf660076f, State: Running, Role: LEADER
I20260929 22:50:19.399655 16566 consensus_queue.cc:237] T 409d925c4ae143a5be051496c7d2f153 P 2bcc9256a4f84ef19cdb214bf660076f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e77842f330e4c0c9797c75ddf7a706a" member_type: VOTER last_known_addr { host: "127.14.147.193" port: 45367 } } peers { permanent_uuid: "2bcc9256a4f84ef19cdb214bf660076f" member_type: VOTER last_known_addr { host: "127.14.147.194" port: 46111 } } peers { permanent_uuid: "3a977c79b99544e08ef536df3e5fef57" member_type: VOTER last_known_addr { host: "127.14.147.195" port: 44731 } }
I20260929 22:50:19.399896 16552 tablet_replica.cc:439] T 409d925c4ae143a5be051496c7d2f153 P 2bcc9256a4f84ef19cdb214bf660076f: TxnStatusTablet state changed. Reason: New leader 2bcc9256a4f84ef19cdb214bf660076f. Latest consensus state: current_term: 1 leader_uuid: "2bcc9256a4f84ef19cdb214bf660076f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e77842f330e4c0c9797c75ddf7a706a" member_type: VOTER last_known_addr { host: "127.14.147.193" port: 45367 } } peers { permanent_uuid: "2bcc9256a4f84ef19cdb214bf660076f" member_type: VOTER last_known_addr { host: "127.14.147.194" port: 46111 } } peers { permanent_uuid: "3a977c79b99544e08ef536df3e5fef57" member_type: VOTER last_known_addr { host: "127.14.147.195" port: 44731 } } }
I20260929 22:50:19.399964 16552 tablet_replica.cc:442] T 409d925c4ae143a5be051496c7d2f153 P 2bcc9256a4f84ef19cdb214bf660076f: This TxnStatusTablet replica's current role is: LEADER
I20260929 22:50:19.400110 16586 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260929 22:50:19.400221 16586 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260929 22:50:19.400223 16039 catalog_manager.cc:5495] T 409d925c4ae143a5be051496c7d2f153 P 2bcc9256a4f84ef19cdb214bf660076f reported cstate change: term changed from 0 to 1, leader changed from <none> to 2bcc9256a4f84ef19cdb214bf660076f (127.14.147.194). New cstate: current_term: 1 leader_uuid: "2bcc9256a4f84ef19cdb214bf660076f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e77842f330e4c0c9797c75ddf7a706a" member_type: VOTER last_known_addr { host: "127.14.147.193" port: 45367 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2bcc9256a4f84ef19cdb214bf660076f" member_type: VOTER last_known_addr { host: "127.14.147.194" port: 46111 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3a977c79b99544e08ef536df3e5fef57" member_type: VOTER last_known_addr { host: "127.14.147.195" port: 44731 } health_report { overall_health: UNKNOWN } } }
I20260929 22:50:19.424891 16542 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:50:19.422668 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54300 (local address 127.14.147.194:46111)
0929 22:50:19.422714 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0929 22:50:19.422716 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 22:50:19.422725 (+ 9us) server_negotiation.cc:407] Connection header received
0929 22:50:19.422816 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 22:50:19.422817 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 22:50:19.422829 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 22:50:19.422871 (+ 42us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 22:50:19.422987 (+ 116us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:50:19.423512 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:50:19.424231 (+ 719us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:50:19.424427 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:50:19.424480 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 22:50:19.424646 (+ 166us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 22:50:19.424682 (+ 36us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 22:50:19.424721 (+ 39us) server_negotiation.cc:1035] Waiting for connection context
0929 22:50:19.424781 (+ 60us) server_negotiation.cc:299] Negotiation successful
0929 22:50:19.424807 (+ 26us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260929 22:50:19.424899 16581 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:50:19.422573 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.147.194:46111 (local address 127.0.0.1:54300)
0929 22:50:19.422673 (+ 100us) negotiation.cc:107] Waiting for socket to connect
0929 22:50:19.422686 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 22:50:19.422743 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 22:50:19.422875 (+ 132us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 22:50:19.422876 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 22:50:19.422877 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0929 22:50:19.422936 (+ 59us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 22:50:19.422938 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 22:50:19.423581 (+ 643us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 22:50:19.423582 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 22:50:19.424158 (+ 576us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 22:50:19.424159 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 22:50:19.424487 (+ 328us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 22:50:19.424488 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 22:50:19.424555 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 22:50:19.424571 (+ 16us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0929 22:50:19.424731 (+ 160us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0929 22:50:19.424739 (+ 8us) client_negotiation.cc:769] Sending connection context
0929 22:50:19.424779 (+ 40us) client_negotiation.cc:240] Negotiation successful
0929 22:50:19.424834 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":54}
I20260929 22:50:19.426132 16197 raft_consensus.cc:1261] T 409d925c4ae143a5be051496c7d2f153 P 0e77842f330e4c0c9797c75ddf7a706a [term 1 FOLLOWER]: Refusing update from remote peer 2bcc9256a4f84ef19cdb214bf660076f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 22:50:19.426170 16477 raft_consensus.cc:1261] T 409d925c4ae143a5be051496c7d2f153 P 3a977c79b99544e08ef536df3e5fef57 [term 1 FOLLOWER]: Refusing update from remote peer 2bcc9256a4f84ef19cdb214bf660076f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 22:50:19.426349 16566 consensus_queue.cc:1035] T 409d925c4ae143a5be051496c7d2f153 P 2bcc9256a4f84ef19cdb214bf660076f [LEADER]: Connected to new peer: Peer: permanent_uuid: "0e77842f330e4c0c9797c75ddf7a706a" member_type: VOTER last_known_addr { host: "127.14.147.193" port: 45367 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 22:50:19.426473 16566 consensus_queue.cc:1035] T 409d925c4ae143a5be051496c7d2f153 P 2bcc9256a4f84ef19cdb214bf660076f [LEADER]: Connected to new peer: Peer: permanent_uuid: "3a977c79b99544e08ef536df3e5fef57" member_type: VOTER last_known_addr { host: "127.14.147.195" port: 44731 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 22:50:19.426988 16554 tablet_replica.cc:439] T 409d925c4ae143a5be051496c7d2f153 P 0e77842f330e4c0c9797c75ddf7a706a: TxnStatusTablet state changed. Reason: New leader 2bcc9256a4f84ef19cdb214bf660076f. Latest consensus state: current_term: 1 leader_uuid: "2bcc9256a4f84ef19cdb214bf660076f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e77842f330e4c0c9797c75ddf7a706a" member_type: VOTER last_known_addr { host: "127.14.147.193" port: 45367 } } peers { permanent_uuid: "2bcc9256a4f84ef19cdb214bf660076f" member_type: VOTER last_known_addr { host: "127.14.147.194" port: 46111 } } peers { permanent_uuid: "3a977c79b99544e08ef536df3e5fef57" member_type: VOTER last_known_addr { host: "127.14.147.195" port: 44731 } } }
I20260929 22:50:19.427079 16554 tablet_replica.cc:442] T 409d925c4ae143a5be051496c7d2f153 P 0e77842f330e4c0c9797c75ddf7a706a: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 22:50:19.427001 16553 tablet_replica.cc:439] T 409d925c4ae143a5be051496c7d2f153 P 3a977c79b99544e08ef536df3e5fef57: TxnStatusTablet state changed. Reason: New leader 2bcc9256a4f84ef19cdb214bf660076f. Latest consensus state: current_term: 1 leader_uuid: "2bcc9256a4f84ef19cdb214bf660076f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e77842f330e4c0c9797c75ddf7a706a" member_type: VOTER last_known_addr { host: "127.14.147.193" port: 45367 } } peers { permanent_uuid: "2bcc9256a4f84ef19cdb214bf660076f" member_type: VOTER last_known_addr { host: "127.14.147.194" port: 46111 } } peers { permanent_uuid: "3a977c79b99544e08ef536df3e5fef57" member_type: VOTER last_known_addr { host: "127.14.147.195" port: 44731 } } }
I20260929 22:50:19.427114 16553 tablet_replica.cc:442] T 409d925c4ae143a5be051496c7d2f153 P 3a977c79b99544e08ef536df3e5fef57: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 22:50:19.427721 16556 tablet_replica.cc:439] T 409d925c4ae143a5be051496c7d2f153 P 2bcc9256a4f84ef19cdb214bf660076f: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "2bcc9256a4f84ef19cdb214bf660076f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e77842f330e4c0c9797c75ddf7a706a" member_type: VOTER last_known_addr { host: "127.14.147.193" port: 45367 } } peers { permanent_uuid: "2bcc9256a4f84ef19cdb214bf660076f" member_type: VOTER last_known_addr { host: "127.14.147.194" port: 46111 } } peers { permanent_uuid: "3a977c79b99544e08ef536df3e5fef57" member_type: VOTER last_known_addr { host: "127.14.147.195" port: 44731 } } }
I20260929 22:50:19.427817 16556 tablet_replica.cc:442] T 409d925c4ae143a5be051496c7d2f153 P 2bcc9256a4f84ef19cdb214bf660076f: This TxnStatusTablet replica's current role is: LEADER
I20260929 22:50:19.427953 16559 mvcc.cc:205] Tried to move back new op lower bound from 7334798210767659008 to 7334798210661281792. Current Snapshot: MvccSnapshot[applied={T|T < 7334798210767659008 or (T in {7334798210767659008})}]
I20260929 22:50:19.428287 16561 mvcc.cc:205] Tried to move back new op lower bound from 7334798210767659008 to 7334798210661281792. Current Snapshot: MvccSnapshot[applied={T|T < 7334798210767659008}]
I20260929 22:50:19.428294 16556 tablet_replica.cc:439] T 409d925c4ae143a5be051496c7d2f153 P 2bcc9256a4f84ef19cdb214bf660076f: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "2bcc9256a4f84ef19cdb214bf660076f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e77842f330e4c0c9797c75ddf7a706a" member_type: VOTER last_known_addr { host: "127.14.147.193" port: 45367 } } peers { permanent_uuid: "2bcc9256a4f84ef19cdb214bf660076f" member_type: VOTER last_known_addr { host: "127.14.147.194" port: 46111 } } peers { permanent_uuid: "3a977c79b99544e08ef536df3e5fef57" member_type: VOTER last_known_addr { host: "127.14.147.195" port: 44731 } } }
I20260929 22:50:19.428383 16556 tablet_replica.cc:442] T 409d925c4ae143a5be051496c7d2f153 P 2bcc9256a4f84ef19cdb214bf660076f: This TxnStatusTablet replica's current role is: LEADER
I20260929 22:50:19.428387 16553 tablet_replica.cc:439] T 409d925c4ae143a5be051496c7d2f153 P 3a977c79b99544e08ef536df3e5fef57: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "2bcc9256a4f84ef19cdb214bf660076f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e77842f330e4c0c9797c75ddf7a706a" member_type: VOTER last_known_addr { host: "127.14.147.193" port: 45367 } } peers { permanent_uuid: "2bcc9256a4f84ef19cdb214bf660076f" member_type: VOTER last_known_addr { host: "127.14.147.194" port: 46111 } } peers { permanent_uuid: "3a977c79b99544e08ef536df3e5fef57" member_type: VOTER last_known_addr { host: "127.14.147.195" port: 44731 } } }
I20260929 22:50:19.428453 16553 tablet_replica.cc:442] T 409d925c4ae143a5be051496c7d2f153 P 3a977c79b99544e08ef536df3e5fef57: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 22:50:19.428840 16554 tablet_replica.cc:439] T 409d925c4ae143a5be051496c7d2f153 P 0e77842f330e4c0c9797c75ddf7a706a: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "2bcc9256a4f84ef19cdb214bf660076f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e77842f330e4c0c9797c75ddf7a706a" member_type: VOTER last_known_addr { host: "127.14.147.193" port: 45367 } } peers { permanent_uuid: "2bcc9256a4f84ef19cdb214bf660076f" member_type: VOTER last_known_addr { host: "127.14.147.194" port: 46111 } } peers { permanent_uuid: "3a977c79b99544e08ef536df3e5fef57" member_type: VOTER last_known_addr { host: "127.14.147.195" port: 44731 } } }
I20260929 22:50:19.428923 16554 tablet_replica.cc:442] T 409d925c4ae143a5be051496c7d2f153 P 0e77842f330e4c0c9797c75ddf7a706a: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 22:50:19.435207 16542 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:50:19.432640 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.147.194:51973 (local address 127.14.147.194:46111)
0929 22:50:19.432681 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0929 22:50:19.432683 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 22:50:19.432803 (+ 120us) server_negotiation.cc:407] Connection header received
0929 22:50:19.432861 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 22:50:19.432861 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 22:50:19.432869 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 22:50:19.432906 (+ 37us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 22:50:19.433018 (+ 112us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:50:19.433474 (+ 456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:50:19.434554 (+ 1080us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:50:19.434738 (+ 184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:50:19.434785 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 22:50:19.434946 (+ 161us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 22:50:19.434973 (+ 27us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 22:50:19.435003 (+ 30us) server_negotiation.cc:1035] Waiting for connection context
0929 22:50:19.435074 (+ 71us) server_negotiation.cc:299] Negotiation successful
0929 22:50:19.435105 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":14,"server-negotiator.queue_time_us":19}
I20260929 22:50:19.435282 16593 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:50:19.432574 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.147.194:46111 (local address 127.14.147.194:51973)
0929 22:50:19.432747 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0929 22:50:19.432757 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0929 22:50:19.432815 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 22:50:19.432918 (+ 103us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 22:50:19.432919 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 22:50:19.432920 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0929 22:50:19.432973 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 22:50:19.432974 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 22:50:19.433531 (+ 557us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 22:50:19.433532 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 22:50:19.434476 (+ 944us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 22:50:19.434478 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 22:50:19.434800 (+ 322us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 22:50:19.434801 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 22:50:19.434858 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 22:50:19.434876 (+ 18us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0929 22:50:19.435015 (+ 139us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0929 22:50:19.435016 (+ 1us) client_negotiation.cc:769] Sending connection context
0929 22:50:19.435070 (+ 54us) client_negotiation.cc:240] Negotiation successful
0929 22:50:19.435130 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":139,"mutex_wait_us":2,"thread_start_us":79,"threads_started":1}
I20260929 22:50:19.437573 16197 raft_consensus.cc:1261] T 76fab0fef8764bdf9655355daf51c565 P 0e77842f330e4c0c9797c75ddf7a706a [term 1 FOLLOWER]: Refusing update from remote peer 2bcc9256a4f84ef19cdb214bf660076f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 22:50:19.437573 16477 raft_consensus.cc:1261] T 76fab0fef8764bdf9655355daf51c565 P 3a977c79b99544e08ef536df3e5fef57 [term 1 FOLLOWER]: Refusing update from remote peer 2bcc9256a4f84ef19cdb214bf660076f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 22:50:19.437770 16566 consensus_queue.cc:1035] T 76fab0fef8764bdf9655355daf51c565 P 2bcc9256a4f84ef19cdb214bf660076f [LEADER]: Connected to new peer: Peer: permanent_uuid: "0e77842f330e4c0c9797c75ddf7a706a" member_type: VOTER last_known_addr { host: "127.14.147.193" port: 45367 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 22:50:19.437887 16566 consensus_queue.cc:1035] T 76fab0fef8764bdf9655355daf51c565 P 2bcc9256a4f84ef19cdb214bf660076f [LEADER]: Connected to new peer: Peer: permanent_uuid: "3a977c79b99544e08ef536df3e5fef57" member_type: VOTER last_known_addr { host: "127.14.147.195" port: 44731 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 22:50:19.449043 16039 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:54240:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 22:50:19.449159 16039 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:54240:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 22:50:19.449700 16039 catalog_manager.cc:5782] T 00000000000000000000000000000000 P d20dfacee9bd43d3b147457b590bc600: Sending DeleteTablet for 3 replicas of tablet 76fab0fef8764bdf9655355daf51c565
I20260929 22:50:19.449941 16175 tablet_service.cc:1514] Processing DeleteTablet for tablet 76fab0fef8764bdf9655355daf51c565 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 22:50:19 UTC) from {username='kudu', principal='kudu/127.14.147.254@KRBTEST.COM'} at 127.0.0.1:59622
I20260929 22:50:19.449949 16457 tablet_service.cc:1514] Processing DeleteTablet for tablet 76fab0fef8764bdf9655355daf51c565 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 22:50:19 UTC) from {username='kudu', principal='kudu/127.14.147.254@KRBTEST.COM'} at 127.0.0.1:50490
I20260929 22:50:19.450078 16571 tablet_replica.cc:330] T 76fab0fef8764bdf9655355daf51c565 P 3a977c79b99544e08ef536df3e5fef57: stopping tablet replica
I20260929 22:50:19.450132 16571 raft_consensus.cc:2229] T 76fab0fef8764bdf9655355daf51c565 P 3a977c79b99544e08ef536df3e5fef57 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 22:50:19.450165 16571 raft_consensus.cc:2258] T 76fab0fef8764bdf9655355daf51c565 P 3a977c79b99544e08ef536df3e5fef57 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 22:50:19.450513 16570 tablet_replica.cc:330] T 76fab0fef8764bdf9655355daf51c565 P 0e77842f330e4c0c9797c75ddf7a706a: stopping tablet replica
I20260929 22:50:19.450575 16570 raft_consensus.cc:2229] T 76fab0fef8764bdf9655355daf51c565 P 0e77842f330e4c0c9797c75ddf7a706a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 22:50:19.450575 16571 ts_tablet_manager.cc:1905] T 76fab0fef8764bdf9655355daf51c565 P 3a977c79b99544e08ef536df3e5fef57: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 22:50:19.450611 16570 raft_consensus.cc:2258] T 76fab0fef8764bdf9655355daf51c565 P 0e77842f330e4c0c9797c75ddf7a706a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 22:50:19.450944 16570 ts_tablet_manager.cc:1905] T 76fab0fef8764bdf9655355daf51c565 P 0e77842f330e4c0c9797c75ddf7a706a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 22:50:19.452318 16316 tablet_service.cc:1514] Processing DeleteTablet for tablet 76fab0fef8764bdf9655355daf51c565 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 22:50:19 UTC) from {username='kudu', principal='kudu/127.14.147.254@KRBTEST.COM'} at 127.0.0.1:54288
I20260929 22:50:19.452466 16569 tablet_replica.cc:330] T 76fab0fef8764bdf9655355daf51c565 P 2bcc9256a4f84ef19cdb214bf660076f: stopping tablet replica
I20260929 22:50:19.452456 16570 ts_tablet_manager.cc:1918] T 76fab0fef8764bdf9655355daf51c565 P 0e77842f330e4c0c9797c75ddf7a706a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260929 22:50:19.452456 16571 ts_tablet_manager.cc:1918] T 76fab0fef8764bdf9655355daf51c565 P 3a977c79b99544e08ef536df3e5fef57: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260929 22:50:19.452512 16570 log.cc:1196] T 76fab0fef8764bdf9655355daf51c565 P 0e77842f330e4c0c9797c75ddf7a706a: Deleting WAL directory at /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790722213368738-14927-0/minicluster-data/ts-0/wal/wals/76fab0fef8764bdf9655355daf51c565
I20260929 22:50:19.452512 16571 log.cc:1196] T 76fab0fef8764bdf9655355daf51c565 P 3a977c79b99544e08ef536df3e5fef57: Deleting WAL directory at /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790722213368738-14927-0/minicluster-data/ts-2/wal/wals/76fab0fef8764bdf9655355daf51c565
I20260929 22:50:19.452510 16569 raft_consensus.cc:2229] T 76fab0fef8764bdf9655355daf51c565 P 2bcc9256a4f84ef19cdb214bf660076f [term 1 LEADER]: Raft consensus shutting down.
I20260929 22:50:19.452582 16569 raft_consensus.cc:2258] T 76fab0fef8764bdf9655355daf51c565 P 2bcc9256a4f84ef19cdb214bf660076f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 22:50:19.452726 16570 ts_tablet_manager.cc:1939] T 76fab0fef8764bdf9655355daf51c565 P 0e77842f330e4c0c9797c75ddf7a706a: Deleting consensus metadata
I20260929 22:50:19.452726 16571 ts_tablet_manager.cc:1939] T 76fab0fef8764bdf9655355daf51c565 P 3a977c79b99544e08ef536df3e5fef57: Deleting consensus metadata
I20260929 22:50:19.453056 16026 catalog_manager.cc:4841] TS 0e77842f330e4c0c9797c75ddf7a706a (127.14.147.193:45367): tablet 76fab0fef8764bdf9655355daf51c565 (table test-table [id=8ca06046ebed43dc8b4c2ddcfae5b0dd]) successfully deleted
I20260929 22:50:19.453161 16025 catalog_manager.cc:4841] TS 3a977c79b99544e08ef536df3e5fef57 (127.14.147.195:44731): tablet 76fab0fef8764bdf9655355daf51c565 (table test-table [id=8ca06046ebed43dc8b4c2ddcfae5b0dd]) successfully deleted
I20260929 22:50:19.453377 16569 ts_tablet_manager.cc:1905] T 76fab0fef8764bdf9655355daf51c565 P 2bcc9256a4f84ef19cdb214bf660076f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 22:50:19.454476 16569 ts_tablet_manager.cc:1918] T 76fab0fef8764bdf9655355daf51c565 P 2bcc9256a4f84ef19cdb214bf660076f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260929 22:50:19.454521 16569 log.cc:1196] T 76fab0fef8764bdf9655355daf51c565 P 2bcc9256a4f84ef19cdb214bf660076f: Deleting WAL directory at /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790722213368738-14927-0/minicluster-data/ts-1/wal/wals/76fab0fef8764bdf9655355daf51c565
I20260929 22:50:19.454694 16569 ts_tablet_manager.cc:1939] T 76fab0fef8764bdf9655355daf51c565 P 2bcc9256a4f84ef19cdb214bf660076f: Deleting consensus metadata
I20260929 22:50:19.454983 16025 catalog_manager.cc:4841] TS 2bcc9256a4f84ef19cdb214bf660076f (127.14.147.194:46111): tablet 76fab0fef8764bdf9655355daf51c565 (table test-table [id=8ca06046ebed43dc8b4c2ddcfae5b0dd]) successfully deleted
Sep 29 22:50:19 dist-test-slave-98mk krb5kdc[15995](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790722218, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.14.147.193@KRBTEST.COM
I20260929 22:50:19.467936 16543 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:50:19.452208 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59624 (local address 127.14.147.193:45367)
0929 22:50:19.452251 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0929 22:50:19.452253 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 22:50:19.459312 (+ 7059us) server_negotiation.cc:407] Connection header received
0929 22:50:19.459348 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 22:50:19.459350 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 22:50:19.459363 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 22:50:19.459471 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0929 22:50:19.460066 (+ 595us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:50:19.460824 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:50:19.461583 (+ 759us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:50:19.461883 (+ 300us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:50:19.461953 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 22:50:19.464921 (+ 2968us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 22:50:19.464936 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 22:50:19.464937 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 22:50:19.464944 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0929 22:50:19.466819 (+ 1875us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 22:50:19.467141 (+ 322us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 22:50:19.467144 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 22:50:19.467145 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 22:50:19.467224 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 22:50:19.467388 (+ 164us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 22:50:19.467389 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 22:50:19.467389 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 22:50:19.467577 (+ 188us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 22:50:19.467715 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0929 22:50:19.467750 (+ 35us) server_negotiation.cc:299] Negotiation successful
0929 22:50:19.467833 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
W20260929 22:50:19.468173 16237 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.SetFlag from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:59624
I20260929 22:50:19.474537 16093 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:50:19.469067 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54264 (local address 127.14.147.254:46225)
0929 22:50:19.469114 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0929 22:50:19.469116 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 22:50:19.469183 (+ 67us) server_negotiation.cc:407] Connection header received
0929 22:50:19.469253 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 22:50:19.469254 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 22:50:19.469265 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 22:50:19.469314 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0929 22:50:19.469763 (+ 449us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:50:19.470271 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:50:19.470987 (+ 716us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:50:19.471180 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:50:19.471234 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 22:50:19.472339 (+ 1105us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 22:50:19.472341 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 22:50:19.472341 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 22:50:19.472348 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0929 22:50:19.473578 (+ 1230us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 22:50:19.473867 (+ 289us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 22:50:19.473868 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 22:50:19.473868 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 22:50:19.473920 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 22:50:19.474076 (+ 156us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 22:50:19.474077 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 22:50:19.474078 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 22:50:19.474187 (+ 109us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 22:50:19.474274 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0929 22:50:19.474368 (+ 94us) server_negotiation.cc:299] Negotiation successful
0929 22:50:19.474446 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
W20260929 22:50:19.475095 16039 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:54264
I20260929 22:50:19.476102 14927 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhWOCLk/build/release/bin/kudu with pid 16112
I20260929 22:50:19.482007 14927 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhWOCLk/build/release/bin/kudu with pid 16252
I20260929 22:50:19.487517 14927 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhWOCLk/build/release/bin/kudu with pid 16392
I20260929 22:50:19.491528 14927 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhWOCLk/build/release/bin/kudu with pid 16008
2026-09-29T22:50:19Z chronyd exiting
[ OK ] SecurityITest.SmokeTestAsAuthorizedUser (2340 ms)
[ RUN ] SecurityITest.TxnSmokeWithDifferentUserTypes
Loading random data
Initializing database '/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790722213368738-14927-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 22:50:19 dist-test-slave-98mk krb5kdc[16603](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 22:50:19 dist-test-slave-98mk krb5kdc[16603](info): set up 2 sockets
Sep 29 22:50:19 dist-test-slave-98mk krb5kdc[16603](info): commencing operation
krb5kdc: starting...
W20260929 22:50:20.439666 14927 mini_kdc.cc:121] Time spent starting KDC: real 0.927s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790722213368738-14927-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790722213368738-14927-0/krb5kdc/test-user.keytab.
Sep 29 22:50:20 dist-test-slave-98mk krb5kdc[16603](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790722220, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T22:50:20Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T22:50:20Z Disabled control of system clock
WARNING: no policy specified for kudu/127.14.147.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.147.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.147.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790722213368738-14927-0/krb5kdc/kudu_127.14.147.254.keytab.
Entry for principal kudu/127.14.147.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790722213368738-14927-0/krb5kdc/kudu_127.14.147.254.keytab.
I20260929 22:50:20.552593 14927 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhWOCLk/build/release/bin/kudu
/tmp/dist-test-taskhWOCLk/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790722213368738-14927-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790722213368738-14927-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790722213368738-14927-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790722213368738-14927-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.147.254:35089
--webserver_interface=127.14.147.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790722213368738-14927-0/krb5kdc/kudu_127.14.147.254.keytab
--principal=kudu/127.14.147.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.147.212:46721
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.147.254:35089
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790722213368738-14927-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790722213368738-14927-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790722213368738-14927-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 22:50:20.632906 16616 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 22:50:20.633145 16616 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 22:50:20.633178 16616 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 22:50:20.634394 16616 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 22:50:20.634439 16616 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260929 22:50:20.634455 16616 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 22:50:20.634472 16616 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 22:50:20.634487 16616 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 22:50:20.634503 16616 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 22:50:20.636065 16616 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.147.212:46721
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790722213368738-14927-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790722213368738-14927-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.14.147.254:35089
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790722213368738-14927-0/krb5kdc/kudu_127.14.147.254.keytab
--principal=kudu/127.14.147.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.147.254:35089
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790722213368738-14927-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.14.147.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16616
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790722213368738-14927-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 21:58:54 UTC on ccycloud.quasar-xguoti.root.comops.site
build id 4681390
I20260929 22:50:20.636329 16616 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 22:50:20.636548 16616 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 22:50:20.641213 16624 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 22:50:20.641259 16616 server_base.cc:1034] running on GCE node
W20260929 22:50:20.641227 16621 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 22:50:20.641230 16622 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 22:50:20.641652 16616 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 22:50:20.641945 16616 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 22:50:20.643090 16616 hybrid_clock.cc:648] HybridClock initialized: now 1790722220643082 us; error 34 us; skew 500 ppm
Sep 29 22:50:20 dist-test-slave-98mk krb5kdc[16603](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790722220, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.147.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 22:50:20.645553 16616 init.cc:377] Logged in from keytab as kudu/127.14.147.254@KRBTEST.COM (short username kudu)
I20260929 22:50:20.645992 16616 webserver.cc:460] Webserver started at http://127.14.147.254:43103/ using document root <none> and password file <none>
I20260929 22:50:20.646214 16616 fs_manager.cc:359] Metadata directory not provided
I20260929 22:50:20.646262 16616 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 22:50:20.646389 16616 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 22:50:20.647750 16616 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790722213368738-14927-0/minicluster-data/master-0/data/instance:
uuid: "e2c35b91e0064597ac06279423fc997d"
format_stamp: "Formatted at 2026-09-29 22:50:20 on dist-test-slave-98mk"
server_key: "8ec25d314cba1b018c524a62327c234b"
server_key_iv: "c0775b07dfd8f4aab324497e2454a682"
server_key_version: "encryptionkey@0"
I20260929 22:50:20.648190 16616 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790722213368738-14927-0/minicluster-data/master-0/wal/instance:
uuid: "e2c35b91e0064597ac06279423fc997d"
format_stamp: "Formatted at 2026-09-29 22:50:20 on dist-test-slave-98mk"
server_key: "8ec25d314cba1b018c524a62327c234b"
server_key_iv: "c0775b07dfd8f4aab324497e2454a682"
server_key_version: "encryptionkey@0"
I20260929 22:50:20.649760 16616 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.001s
I20260929 22:50:20.650592 16631 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 22:50:20.650764 16616 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260929 22:50:20.650841 16616 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790722213368738-14927-0/minicluster-data/master-0/data,/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790722213368738-14927-0/minicluster-data/master-0/wal
uuid: "e2c35b91e0064597ac06279423fc997d"
format_stamp: "Formatted at 2026-09-29 22:50:20 on dist-test-slave-98mk"
server_key: "8ec25d314cba1b018c524a62327c234b"
server_key_iv: "c0775b07dfd8f4aab324497e2454a682"
server_key_version: "encryptionkey@0"
I20260929 22:50:20.650914 16616 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790722213368738-14927-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790722213368738-14927-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790722213368738-14927-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 22:50:20.662739 16616 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 22:50:20.664775 16616 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 22:50:20.664927 16616 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 22:50:20.669243 16616 rpc_server.cc:307] RPC server started. Bound to: 127.14.147.254:35089
I20260929 22:50:20.669323 16694 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.147.254:35089 every 8 connection(s)
I20260929 22:50:20.669607 16616 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790722213368738-14927-0/minicluster-data/master-0/data/info.pb
I20260929 22:50:20.670217 16695 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 22:50:20.672556 16695 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e2c35b91e0064597ac06279423fc997d: Bootstrap starting.
I20260929 22:50:20.673154 16695 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e2c35b91e0064597ac06279423fc997d: Neither blocks nor log segments found. Creating new log.
I20260929 22:50:20.673424 16695 log.cc:824] T 00000000000000000000000000000000 P e2c35b91e0064597ac06279423fc997d: Log is configured to *not* fsync() on all Append() calls
I20260929 22:50:20.674136 16695 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e2c35b91e0064597ac06279423fc997d: No bootstrap required, opened a new log
I20260929 22:50:20.676314 16695 raft_consensus.cc:348] T 00000000000000000000000000000000 P e2c35b91e0064597ac06279423fc997d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e2c35b91e0064597ac06279423fc997d" member_type: VOTER last_known_addr { host: "127.14.147.254" port: 35089 } }
I20260929 22:50:20.676481 16695 raft_consensus.cc:374] T 00000000000000000000000000000000 P e2c35b91e0064597ac06279423fc997d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 22:50:20.676503 16695 raft_consensus.cc:729] T 00000000000000000000000000000000 P e2c35b91e0064597ac06279423fc997d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e2c35b91e0064597ac06279423fc997d, State: Initialized, Role: FOLLOWER
I20260929 22:50:20.676635 16695 consensus_queue.cc:260] T 00000000000000000000000000000000 P e2c35b91e0064597ac06279423fc997d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e2c35b91e0064597ac06279423fc997d" member_type: VOTER last_known_addr { host: "127.14.147.254" port: 35089 } }
I20260929 22:50:20.676714 16695 raft_consensus.cc:388] T 00000000000000000000000000000000 P e2c35b91e0064597ac06279423fc997d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 22:50:20.676743 16695 raft_consensus.cc:482] T 00000000000000000000000000000000 P e2c35b91e0064597ac06279423fc997d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 22:50:20.676779 16695 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e2c35b91e0064597ac06279423fc997d [term 0 FOLLOWER]: Advancing to term 1
I20260929 22:50:20.677460 16695 raft_consensus.cc:504] T 00000000000000000000000000000000 P e2c35b91e0064597ac06279423fc997d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e2c35b91e0064597ac06279423fc997d" member_type: VOTER last_known_addr { host: "127.14.147.254" port: 35089 } }
I20260929 22:50:20.677556 16695 leader_election.cc:302] T 00000000000000000000000000000000 P e2c35b91e0064597ac06279423fc997d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e2c35b91e0064597ac06279423fc997d; no voters:
I20260929 22:50:20.677716 16695 leader_election.cc:288] T 00000000000000000000000000000000 P e2c35b91e0064597ac06279423fc997d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 22:50:20.677758 16698 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e2c35b91e0064597ac06279423fc997d [term 1 FOLLOWER]: Leader election won for term 1
I20260929 22:50:20.677902 16698 raft_consensus.cc:686] T 00000000000000000000000000000000 P e2c35b91e0064597ac06279423fc997d [term 1 LEADER]: Becoming Leader. State: Replica: e2c35b91e0064597ac06279423fc997d, State: Running, Role: LEADER
I20260929 22:50:20.677976 16695 sys_catalog.cc:564] T 00000000000000000000000000000000 P e2c35b91e0064597ac06279423fc997d [sys.catalog]: configured and running, proceeding with master startup.
I20260929 22:50:20.677996 16698 consensus_queue.cc:237] T 00000000000000000000000000000000 P e2c35b91e0064597ac06279423fc997d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e2c35b91e0064597ac06279423fc997d" member_type: VOTER last_known_addr { host: "127.14.147.254" port: 35089 } }
I20260929 22:50:20.678347 16699 sys_catalog.cc:455] T 00000000000000000000000000000000 P e2c35b91e0064597ac06279423fc997d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e2c35b91e0064597ac06279423fc997d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e2c35b91e0064597ac06279423fc997d" member_type: VOTER last_known_addr { host: "127.14.147.254" port: 35089 } } }
I20260929 22:50:20.678375 16700 sys_catalog.cc:455] T 00000000000000000000000000000000 P e2c35b91e0064597ac06279423fc997d [sys.catalog]: SysCatalogTable state changed. Reason: New leader e2c35b91e0064597ac06279423fc997d. Latest consensus state: current_term: 1 leader_uuid: "e2c35b91e0064597ac06279423fc997d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e2c35b91e0064597ac06279423fc997d" member_type: VOTER last_known_addr { host: "127.14.147.254" port: 35089 } } }
I20260929 22:50:20.678527 16700 sys_catalog.cc:458] T 00000000000000000000000000000000 P e2c35b91e0064597ac06279423fc997d [sys.catalog]: This master's current role is: LEADER
I20260929 22:50:20.678642 14927 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhWOCLk/build/release/bin/kudu as pid 16616
I20260929 22:50:20.678722 14927 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790722213368738-14927-0/minicluster-data/master-0/wal/instance
I20260929 22:50:20.678900 14927 external_mini_cluster.cc:1428] Setting key a4e8771b6690312ba678604818560961
I20260929 22:50:20.678972 16707 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 22:50:20.679023 16699 sys_catalog.cc:458] T 00000000000000000000000000000000 P e2c35b91e0064597ac06279423fc997d [sys.catalog]: This master's current role is: LEADER
I20260929 22:50:20.679560 16707 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 22:50:20.682590 16707 catalog_manager.cc:1347] Generated new cluster ID: dbad12ad4c6345d4a87ad99f2fdc5104
I20260929 22:50:20.682654 16707 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 29 22:50:20 dist-test-slave-98mk krb5kdc[16603](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790722220, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.147.254@KRBTEST.COM
I20260929 22:50:20.690196 16716 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:50:20.679472 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40022 (local address 127.14.147.254:35089)
0929 22:50:20.679671 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0929 22:50:20.679681 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 22:50:20.679713 (+ 32us) server_negotiation.cc:407] Connection header received
0929 22:50:20.679834 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 22:50:20.679841 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 22:50:20.679898 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 22:50:20.680042 (+ 144us) server_negotiation.cc:226] Negotiated authn=SASL
0929 22:50:20.680723 (+ 681us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:50:20.682080 (+ 1357us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:50:20.683042 (+ 962us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:50:20.683735 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:50:20.683847 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 22:50:20.686595 (+ 2748us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 22:50:20.686601 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 22:50:20.686601 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 22:50:20.686614 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0929 22:50:20.688548 (+ 1934us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 22:50:20.688945 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 22:50:20.688947 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 22:50:20.688948 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 22:50:20.689059 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 22:50:20.689254 (+ 195us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 22:50:20.689255 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 22:50:20.689255 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 22:50:20.689559 (+ 304us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 22:50:20.689703 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0929 22:50:20.689808 (+ 105us) server_negotiation.cc:299] Negotiation successful
0929 22:50:20.689944 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":150,"thread_start_us":78,"threads_started":1}
I20260929 22:50:20.697086 16707 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 22:50:20.697609 16707 catalog_manager.cc:1504] Loading token signing keys...
I20260929 22:50:20.703399 16707 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e2c35b91e0064597ac06279423fc997d: Generated new TSK 0
I20260929 22:50:20.703644 16707 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.14.147.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.147.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.147.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790722213368738-14927-0/krb5kdc/kudu_127.14.147.193.keytab.
Entry for principal kudu/127.14.147.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790722213368738-14927-0/krb5kdc/kudu_127.14.147.193.keytab.
I20260929 22:50:20.730140 14927 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhWOCLk/build/release/bin/kudu
/tmp/dist-test-taskhWOCLk/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790722213368738-14927-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790722213368738-14927-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790722213368738-14927-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790722213368738-14927-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.147.193:0
--local_ip_for_outbound_sockets=127.14.147.193
--webserver_interface=127.14.147.193
--webserver_port=0
--tserver_master_addrs=127.14.147.254:35089
--keytab_file=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790722213368738-14927-0/krb5kdc/kudu_127.14.147.193.keytab
--principal=kudu/127.14.147.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.147.212:46721
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790722213368738-14927-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790722213368738-14927-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790722213368738-14927-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 22:50:20.812726 16721 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 22:50:20.812938 16721 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 22:50:20.812964 16721 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 22:50:20.814164 16721 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 22:50:20.814205 16721 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 22:50:20.814226 16721 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260929 22:50:20.814239 16721 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260929 22:50:20.814261 16721 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.147.193
I20260929 22:50:20.815873 16721 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.147.212:46721
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790722213368738-14927-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790722213368738-14927-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790722213368738-14927-0/krb5kdc/kudu_127.14.147.193.keytab
--principal=kudu/127.14.147.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.147.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790722213368738-14927-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.14.147.193
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.14.147.254:35089
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16721
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.147.193
--log_dir=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790722213368738-14927-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 21:58:54 UTC on ccycloud.quasar-xguoti.root.comops.site
build id 4681390
I20260929 22:50:20.816089 16721 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 22:50:20.816280 16721 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 22:50:20.821080 16729 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 22:50:20.821096 16727 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 22:50:20.821080 16726 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 22:50:20.821435 16721 server_base.cc:1034] running on GCE node
I20260929 22:50:20.821615 16721 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 22:50:20.821842 16721 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 22:50:20.822971 16721 hybrid_clock.cc:648] HybridClock initialized: now 1790722220822963 us; error 29 us; skew 500 ppm
Sep 29 22:50:20 dist-test-slave-98mk krb5kdc[16603](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790722220, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.147.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 22:50:20.825387 16721 init.cc:377] Logged in from keytab as kudu/127.14.147.193@KRBTEST.COM (short username kudu)
I20260929 22:50:20.825809 16721 webserver.cc:460] Webserver started at http://127.14.147.193:36897/ using document root <none> and password file <none>
I20260929 22:50:20.826027 16721 fs_manager.cc:359] Metadata directory not provided
I20260929 22:50:20.826076 16721 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 22:50:20.826207 16721 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 22:50:20.827263 16721 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790722213368738-14927-0/minicluster-data/ts-0/data/instance:
uuid: "b3941f6d62734fa59b0f1005b814f025"
format_stamp: "Formatted at 2026-09-29 22:50:20 on dist-test-slave-98mk"
server_key: "ac7043c5996ac70593bb9134b37b61a0"
server_key_iv: "6874a75a061f6210487543c5aff79277"
server_key_version: "encryptionkey@0"
I20260929 22:50:20.827620 16721 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790722213368738-14927-0/minicluster-data/ts-0/wal/instance:
uuid: "b3941f6d62734fa59b0f1005b814f025"
format_stamp: "Formatted at 2026-09-29 22:50:20 on dist-test-slave-98mk"
server_key: "ac7043c5996ac70593bb9134b37b61a0"
server_key_iv: "6874a75a061f6210487543c5aff79277"
server_key_version: "encryptionkey@0"
I20260929 22:50:20.828860 16721 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260929 22:50:20.829515 16736 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 22:50:20.829699 16721 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260929 22:50:20.829780 16721 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790722213368738-14927-0/minicluster-data/ts-0/data,/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790722213368738-14927-0/minicluster-data/ts-0/wal
uuid: "b3941f6d62734fa59b0f1005b814f025"
format_stamp: "Formatted at 2026-09-29 22:50:20 on dist-test-slave-98mk"
server_key: "ac7043c5996ac70593bb9134b37b61a0"
server_key_iv: "6874a75a061f6210487543c5aff79277"
server_key_version: "encryptionkey@0"
I20260929 22:50:20.829842 16721 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790722213368738-14927-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790722213368738-14927-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790722213368738-14927-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 22:50:20.845681 16721 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 22:50:20.847903 16721 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 22:50:20.848050 16721 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 22:50:20.848574 16721 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 22:50:20.848618 16721 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 22:50:20.848649 16721 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 22:50:20.848664 16721 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 29 22:50:20 dist-test-slave-98mk krb5kdc[16603](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790722220, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.147.193@KRBTEST.COM for kudu/127.14.147.254@KRBTEST.COM
I20260929 22:50:20.855605 16721 rpc_server.cc:307] RPC server started. Bound to: 127.14.147.193:33765
I20260929 22:50:20.855736 16851 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.147.193:33765 every 8 connection(s)
I20260929 22:50:20.856027 16721 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790722213368738-14927-0/minicluster-data/ts-0/data/info.pb
I20260929 22:50:20.858315 16716 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:50:20.849054 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.147.193:35499 (local address 127.14.147.254:35089)
0929 22:50:20.849118 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0929 22:50:20.849121 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 22:50:20.849322 (+ 201us) server_negotiation.cc:407] Connection header received
0929 22:50:20.849822 (+ 500us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 22:50:20.849824 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 22:50:20.849842 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 22:50:20.849911 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0929 22:50:20.850784 (+ 873us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:50:20.851581 (+ 797us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:50:20.852692 (+ 1111us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:50:20.852905 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:50:20.852963 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 22:50:20.855826 (+ 2863us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 22:50:20.855828 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 22:50:20.855828 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 22:50:20.855837 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0929 22:50:20.857178 (+ 1341us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 22:50:20.857569 (+ 391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 22:50:20.857570 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 22:50:20.857570 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 22:50:20.857626 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 22:50:20.857812 (+ 186us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 22:50:20.857813 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 22:50:20.857813 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 22:50:20.857915 (+ 102us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 22:50:20.857960 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0929 22:50:20.858178 (+ 218us) server_negotiation.cc:299] Negotiation successful
0929 22:50:20.858246 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260929 22:50:20.858500 16745 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:50:20.849023 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.147.254:35089 (local address 127.14.147.193:35499)
0929 22:50:20.849188 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0929 22:50:20.849230 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0929 22:50:20.849726 (+ 496us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 22:50:20.850000 (+ 274us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 22:50:20.850004 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 22:50:20.850361 (+ 357us) client_negotiation.cc:189] Negotiated authn=SASL
0929 22:50:20.850673 (+ 312us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 22:50:20.850680 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 22:50:20.851691 (+ 1011us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 22:50:20.851693 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 22:50:20.852593 (+ 900us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 22:50:20.852597 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 22:50:20.852999 (+ 402us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 22:50:20.853000 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 22:50:20.853127 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 22:50:20.853689 (+ 562us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 22:50:20.853699 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0929 22:50:20.855711 (+ 2012us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 22:50:20.857256 (+ 1545us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 22:50:20.857263 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 22:50:20.857265 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 22:50:20.857505 (+ 240us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 22:50:20.857672 (+ 167us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 22:50:20.857673 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 22:50:20.857673 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 22:50:20.857769 (+ 96us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 22:50:20.857955 (+ 186us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 22:50:20.857959 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 22:50:20.858083 (+ 124us) client_negotiation.cc:769] Sending connection context
0929 22:50:20.858164 (+ 81us) client_negotiation.cc:240] Negotiation successful
0929 22:50:20.858294 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":89,"threads_started":1}
I20260929 22:50:20.858968 16852 heartbeater.cc:347] Connected to a master server at 127.14.147.254:35089
I20260929 22:50:20.859048 16852 heartbeater.cc:464] Registering TS with master...
I20260929 22:50:20.859225 16852 heartbeater.cc:511] Master 127.14.147.254:35089 requested a full tablet report, sending...
I20260929 22:50:20.859755 16647 ts_manager.cc:194] Registered new tserver with Master: b3941f6d62734fa59b0f1005b814f025 (127.14.147.193:33765)
I20260929 22:50:20.860483 16647 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.147.193@KRBTEST.COM'} at 127.14.147.193:35499
I20260929 22:50:20.865329 16858 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:50:20.860004 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.147.254:35089 (local address 127.14.147.193:48153)
0929 22:50:20.860148 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0929 22:50:20.860159 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0929 22:50:20.860215 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 22:50:20.860344 (+ 129us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 22:50:20.860345 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 22:50:20.860512 (+ 167us) client_negotiation.cc:189] Negotiated authn=SASL
0929 22:50:20.860566 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 22:50:20.860568 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 22:50:20.861445 (+ 877us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 22:50:20.861445 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 22:50:20.861961 (+ 516us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 22:50:20.861962 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 22:50:20.862291 (+ 329us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 22:50:20.862291 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 22:50:20.862347 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 22:50:20.862722 (+ 375us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 22:50:20.862725 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0929 22:50:20.863007 (+ 282us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 22:50:20.864414 (+ 1407us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 22:50:20.864415 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 22:50:20.864416 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 22:50:20.864576 (+ 160us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 22:50:20.864767 (+ 191us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 22:50:20.864768 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 22:50:20.864768 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 22:50:20.864838 (+ 70us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 22:50:20.865077 (+ 239us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 22:50:20.865077 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 22:50:20.865118 (+ 41us) client_negotiation.cc:769] Sending connection context
0929 22:50:20.865163 (+ 45us) client_negotiation.cc:240] Negotiation successful
0929 22:50:20.865254 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":81,"threads_started":1}
I20260929 22:50:20.865352 16716 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:50:20.860048 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.147.193:48153 (local address 127.14.147.254:35089)
0929 22:50:20.860088 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0929 22:50:20.860090 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 22:50:20.860212 (+ 122us) server_negotiation.cc:407] Connection header received
0929 22:50:20.860270 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 22:50:20.860271 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 22:50:20.860281 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 22:50:20.860327 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0929 22:50:20.860675 (+ 348us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:50:20.861349 (+ 674us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:50:20.862041 (+ 692us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:50:20.862234 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:50:20.862287 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 22:50:20.863085 (+ 798us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 22:50:20.863086 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 22:50:20.863086 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 22:50:20.863093 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0929 22:50:20.864334 (+ 1241us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 22:50:20.864633 (+ 299us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 22:50:20.864633 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 22:50:20.864634 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 22:50:20.864679 (+ 45us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 22:50:20.864906 (+ 227us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 22:50:20.864907 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 22:50:20.864907 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 22:50:20.865026 (+ 119us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 22:50:20.865078 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0929 22:50:20.865179 (+ 101us) server_negotiation.cc:299] Negotiation successful
0929 22:50:20.865265 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260929 22:50:20.865444 14927 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhWOCLk/build/release/bin/kudu as pid 16721
I20260929 22:50:20.865535 14927 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790722213368738-14927-0/minicluster-data/ts-0/wal/instance
I20260929 22:50:20.865720 14927 external_mini_cluster.cc:1428] Setting key 865a69efb340ed2fb991bb1e99514b8a
WARNING: no policy specified for kudu/127.14.147.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.147.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.147.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790722213368738-14927-0/krb5kdc/kudu_127.14.147.194.keytab.
Entry for principal kudu/127.14.147.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790722213368738-14927-0/krb5kdc/kudu_127.14.147.194.keytab.
I20260929 22:50:20.888425 14927 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhWOCLk/build/release/bin/kudu
/tmp/dist-test-taskhWOCLk/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790722213368738-14927-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790722213368738-14927-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790722213368738-14927-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790722213368738-14927-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.147.194:0
--local_ip_for_outbound_sockets=127.14.147.194
--webserver_interface=127.14.147.194
--webserver_port=0
--tserver_master_addrs=127.14.147.254:35089
--keytab_file=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790722213368738-14927-0/krb5kdc/kudu_127.14.147.194.keytab
--principal=kudu/127.14.147.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.147.212:46721
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790722213368738-14927-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790722213368738-14927-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790722213368738-14927-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 22:50:20.972759 16861 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 22:50:20.972985 16861 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 22:50:20.973029 16861 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 22:50:20.974253 16861 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 22:50:20.974300 16861 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 22:50:20.974329 16861 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260929 22:50:20.974344 16861 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260929 22:50:20.974368 16861 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.147.194
I20260929 22:50:20.975975 16861 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.147.212:46721
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790722213368738-14927-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790722213368738-14927-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790722213368738-14927-0/krb5kdc/kudu_127.14.147.194.keytab
--principal=kudu/127.14.147.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.147.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790722213368738-14927-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.14.147.194
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.14.147.254:35089
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16861
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.147.194
--log_dir=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790722213368738-14927-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 21:58:54 UTC on ccycloud.quasar-xguoti.root.comops.site
build id 4681390
I20260929 22:50:20.976243 16861 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 22:50:20.976464 16861 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 22:50:20.981428 16869 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 22:50:20.981459 16867 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 22:50:20.981568 16861 server_base.cc:1034] running on GCE node
W20260929 22:50:20.981782 16866 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 22:50:20.982020 16861 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 22:50:20.982312 16861 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 22:50:20.983460 16861 hybrid_clock.cc:648] HybridClock initialized: now 1790722220983451 us; error 26 us; skew 500 ppm
Sep 29 22:50:20 dist-test-slave-98mk krb5kdc[16603](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790722220, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.147.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 22:50:20.985952 16861 init.cc:377] Logged in from keytab as kudu/127.14.147.194@KRBTEST.COM (short username kudu)
I20260929 22:50:20.986439 16861 webserver.cc:460] Webserver started at http://127.14.147.194:33519/ using document root <none> and password file <none>
I20260929 22:50:20.986656 16861 fs_manager.cc:359] Metadata directory not provided
I20260929 22:50:20.986706 16861 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 22:50:20.986826 16861 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 22:50:20.987900 16861 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790722213368738-14927-0/minicluster-data/ts-1/data/instance:
uuid: "cb60c60cc23349ab9e59004d1d2983fa"
format_stamp: "Formatted at 2026-09-29 22:50:20 on dist-test-slave-98mk"
server_key: "e3978f9303c9891c44fe39edc564907d"
server_key_iv: "4365d689d90db112be452ec163c57dd6"
server_key_version: "encryptionkey@0"
I20260929 22:50:20.988227 16861 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790722213368738-14927-0/minicluster-data/ts-1/wal/instance:
uuid: "cb60c60cc23349ab9e59004d1d2983fa"
format_stamp: "Formatted at 2026-09-29 22:50:20 on dist-test-slave-98mk"
server_key: "e3978f9303c9891c44fe39edc564907d"
server_key_iv: "4365d689d90db112be452ec163c57dd6"
server_key_version: "encryptionkey@0"
I20260929 22:50:20.989636 16861 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260929 22:50:20.991065 16876 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 22:50:20.991369 16861 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.002s sys 0.000s
I20260929 22:50:20.991500 16861 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790722213368738-14927-0/minicluster-data/ts-1/data,/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790722213368738-14927-0/minicluster-data/ts-1/wal
uuid: "cb60c60cc23349ab9e59004d1d2983fa"
format_stamp: "Formatted at 2026-09-29 22:50:20 on dist-test-slave-98mk"
server_key: "e3978f9303c9891c44fe39edc564907d"
server_key_iv: "4365d689d90db112be452ec163c57dd6"
server_key_version: "encryptionkey@0"
I20260929 22:50:20.991585 16861 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790722213368738-14927-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790722213368738-14927-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790722213368738-14927-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 22:50:21.011340 16861 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 22:50:21.013859 16861 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 22:50:21.014055 16861 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 22:50:21.014822 16861 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 22:50:21.014881 16861 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 22:50:21.014915 16861 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 22:50:21.014931 16861 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 29 22:50:21 dist-test-slave-98mk krb5kdc[16603](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790722220, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.147.194@KRBTEST.COM for kudu/127.14.147.254@KRBTEST.COM
I20260929 22:50:21.021754 16861 rpc_server.cc:307] RPC server started. Bound to: 127.14.147.194:39385
I20260929 22:50:21.021840 16991 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.147.194:39385 every 8 connection(s)
I20260929 22:50:21.022135 16861 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790722213368738-14927-0/minicluster-data/ts-1/data/info.pb
I20260929 22:50:21.024114 14927 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhWOCLk/build/release/bin/kudu as pid 16861
I20260929 22:50:21.024219 14927 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790722213368738-14927-0/minicluster-data/ts-1/wal/instance
I20260929 22:50:21.024415 14927 external_mini_cluster.cc:1428] Setting key c9bda5b929e3a3366ed413c7ef4eba57
I20260929 22:50:21.024477 16716 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:50:21.015215 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.147.194:59469 (local address 127.14.147.254:35089)
0929 22:50:21.015283 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0929 22:50:21.015285 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 22:50:21.015500 (+ 215us) server_negotiation.cc:407] Connection header received
0929 22:50:21.015997 (+ 497us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 22:50:21.015998 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 22:50:21.016012 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 22:50:21.016094 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0929 22:50:21.016753 (+ 659us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:50:21.017249 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:50:21.018358 (+ 1109us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:50:21.018577 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:50:21.018632 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 22:50:21.021693 (+ 3061us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 22:50:21.021695 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 22:50:21.021695 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 22:50:21.021704 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0929 22:50:21.023095 (+ 1391us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 22:50:21.023506 (+ 411us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 22:50:21.023507 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 22:50:21.023508 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 22:50:21.023567 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 22:50:21.023767 (+ 200us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 22:50:21.023768 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 22:50:21.023768 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 22:50:21.023876 (+ 108us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 22:50:21.023921 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0929 22:50:21.024331 (+ 410us) server_negotiation.cc:299] Negotiation successful
0929 22:50:21.024400 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260929 22:50:21.024704 16885 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:50:21.015194 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.147.254:35089 (local address 127.14.147.194:59469)
0929 22:50:21.015346 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0929 22:50:21.015390 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0929 22:50:21.015903 (+ 513us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 22:50:21.016151 (+ 248us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 22:50:21.016155 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 22:50:21.016422 (+ 267us) client_negotiation.cc:189] Negotiated authn=SASL
0929 22:50:21.016650 (+ 228us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 22:50:21.016655 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 22:50:21.017335 (+ 680us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 22:50:21.017337 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 22:50:21.018272 (+ 935us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 22:50:21.018275 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 22:50:21.018649 (+ 374us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 22:50:21.018650 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 22:50:21.018780 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 22:50:21.019328 (+ 548us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 22:50:21.019342 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0929 22:50:21.021599 (+ 2257us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 22:50:21.023176 (+ 1577us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 22:50:21.023182 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 22:50:21.023184 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 22:50:21.023405 (+ 221us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 22:50:21.023616 (+ 211us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 22:50:21.023616 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 22:50:21.023616 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 22:50:21.023716 (+ 100us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 22:50:21.023976 (+ 260us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 22:50:21.023984 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 22:50:21.024169 (+ 185us) client_negotiation.cc:769] Sending connection context
0929 22:50:21.024288 (+ 119us) client_negotiation.cc:240] Negotiation successful
0929 22:50:21.024477 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":82,"threads_started":1}
I20260929 22:50:21.025189 16992 heartbeater.cc:347] Connected to a master server at 127.14.147.254:35089
I20260929 22:50:21.025275 16992 heartbeater.cc:464] Registering TS with master...
I20260929 22:50:21.025452 16992 heartbeater.cc:511] Master 127.14.147.254:35089 requested a full tablet report, sending...
I20260929 22:50:21.025784 16647 ts_manager.cc:194] Registered new tserver with Master: cb60c60cc23349ab9e59004d1d2983fa (127.14.147.194:39385)
I20260929 22:50:21.026235 16647 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.147.194@KRBTEST.COM'} at 127.14.147.194:59469
I20260929 22:50:21.031879 16998 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:50:21.026162 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.147.254:35089 (local address 127.14.147.194:37061)
0929 22:50:21.026285 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0929 22:50:21.026295 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0929 22:50:21.026345 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 22:50:21.026527 (+ 182us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 22:50:21.026528 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 22:50:21.026730 (+ 202us) client_negotiation.cc:189] Negotiated authn=SASL
0929 22:50:21.026799 (+ 69us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 22:50:21.026801 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 22:50:21.027544 (+ 743us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 22:50:21.027545 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 22:50:21.028060 (+ 515us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 22:50:21.028061 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 22:50:21.028496 (+ 435us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 22:50:21.028496 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 22:50:21.028553 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 22:50:21.028976 (+ 423us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 22:50:21.028981 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0929 22:50:21.029266 (+ 285us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 22:50:21.030840 (+ 1574us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 22:50:21.030841 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 22:50:21.030841 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 22:50:21.031024 (+ 183us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 22:50:21.031220 (+ 196us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 22:50:21.031220 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 22:50:21.031221 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 22:50:21.031268 (+ 47us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 22:50:21.031620 (+ 352us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 22:50:21.031621 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 22:50:21.031662 (+ 41us) client_negotiation.cc:769] Sending connection context
0929 22:50:21.031713 (+ 51us) client_negotiation.cc:240] Negotiation successful
0929 22:50:21.031798 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":92,"thread_start_us":64,"threads_started":1}
I20260929 22:50:21.031914 16716 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:50:21.026345 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.147.194:37061 (local address 127.14.147.254:35089)
0929 22:50:21.026385 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0929 22:50:21.026386 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0929 22:50:21.026394 (+ 8us) server_negotiation.cc:407] Connection header received
0929 22:50:21.026413 (+ 19us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 22:50:21.026414 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 22:50:21.026423 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 22:50:21.026469 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0929 22:50:21.026883 (+ 414us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:50:21.027471 (+ 588us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:50:21.028144 (+ 673us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:50:21.028431 (+ 287us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:50:21.028500 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 22:50:21.029353 (+ 853us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 22:50:21.029354 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 22:50:21.029354 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 22:50:21.029362 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0929 22:50:21.030729 (+ 1367us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 22:50:21.031097 (+ 368us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 22:50:21.031099 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 22:50:21.031099 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 22:50:21.031158 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 22:50:21.031317 (+ 159us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 22:50:21.031317 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 22:50:21.031318 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 22:50:21.031556 (+ 238us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 22:50:21.031631 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0929 22:50:21.031741 (+ 110us) server_negotiation.cc:299] Negotiation successful
0929 22:50:21.031818 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
WARNING: no policy specified for kudu/127.14.147.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.147.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.147.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790722213368738-14927-0/krb5kdc/kudu_127.14.147.195.keytab.
Entry for principal kudu/127.14.147.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790722213368738-14927-0/krb5kdc/kudu_127.14.147.195.keytab.
I20260929 22:50:21.050261 14927 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhWOCLk/build/release/bin/kudu
/tmp/dist-test-taskhWOCLk/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790722213368738-14927-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790722213368738-14927-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790722213368738-14927-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790722213368738-14927-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.147.195:0
--local_ip_for_outbound_sockets=127.14.147.195
--webserver_interface=127.14.147.195
--webserver_port=0
--tserver_master_addrs=127.14.147.254:35089
--keytab_file=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790722213368738-14927-0/krb5kdc/kudu_127.14.147.195.keytab
--principal=kudu/127.14.147.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.147.212:46721
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790722213368738-14927-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790722213368738-14927-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790722213368738-14927-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 22:50:21.130208 17001 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 22:50:21.130424 17001 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 22:50:21.130451 17001 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 22:50:21.131685 17001 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 22:50:21.131726 17001 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 22:50:21.131745 17001 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260929 22:50:21.131757 17001 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260929 22:50:21.131788 17001 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.147.195
I20260929 22:50:21.133345 17001 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.147.212:46721
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790722213368738-14927-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790722213368738-14927-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790722213368738-14927-0/krb5kdc/kudu_127.14.147.195.keytab
--principal=kudu/127.14.147.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.147.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790722213368738-14927-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.14.147.195
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.14.147.254:35089
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17001
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.147.195
--log_dir=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790722213368738-14927-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 21:58:54 UTC on ccycloud.quasar-xguoti.root.comops.site
build id 4681390
I20260929 22:50:21.133576 17001 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 22:50:21.133802 17001 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 22:50:21.138588 17006 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 22:50:21.138576 17007 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 22:50:21.138572 17009 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 22:50:21.138811 17001 server_base.cc:1034] running on GCE node
I20260929 22:50:21.139008 17001 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 22:50:21.139227 17001 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 22:50:21.140359 17001 hybrid_clock.cc:648] HybridClock initialized: now 1790722221140344 us; error 30 us; skew 500 ppm
Sep 29 22:50:21 dist-test-slave-98mk krb5kdc[16603](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790722221, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.147.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 22:50:21.143057 17001 init.cc:377] Logged in from keytab as kudu/127.14.147.195@KRBTEST.COM (short username kudu)
I20260929 22:50:21.143527 17001 webserver.cc:460] Webserver started at http://127.14.147.195:38999/ using document root <none> and password file <none>
I20260929 22:50:21.143723 17001 fs_manager.cc:359] Metadata directory not provided
I20260929 22:50:21.143764 17001 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 22:50:21.143859 17001 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 22:50:21.144884 17001 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790722213368738-14927-0/minicluster-data/ts-2/data/instance:
uuid: "306a909ee2e344a49f9e7691e2ec8ebe"
format_stamp: "Formatted at 2026-09-29 22:50:21 on dist-test-slave-98mk"
server_key: "7c60c380072a826c7871bce7f354a758"
server_key_iv: "317ffff8611abd270f2b9f1111e6db41"
server_key_version: "encryptionkey@0"
I20260929 22:50:21.145175 17001 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790722213368738-14927-0/minicluster-data/ts-2/wal/instance:
uuid: "306a909ee2e344a49f9e7691e2ec8ebe"
format_stamp: "Formatted at 2026-09-29 22:50:21 on dist-test-slave-98mk"
server_key: "7c60c380072a826c7871bce7f354a758"
server_key_iv: "317ffff8611abd270f2b9f1111e6db41"
server_key_version: "encryptionkey@0"
I20260929 22:50:21.146453 17001 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260929 22:50:21.147202 17016 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 22:50:21.147379 17001 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260929 22:50:21.147495 17001 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790722213368738-14927-0/minicluster-data/ts-2/data,/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790722213368738-14927-0/minicluster-data/ts-2/wal
uuid: "306a909ee2e344a49f9e7691e2ec8ebe"
format_stamp: "Formatted at 2026-09-29 22:50:21 on dist-test-slave-98mk"
server_key: "7c60c380072a826c7871bce7f354a758"
server_key_iv: "317ffff8611abd270f2b9f1111e6db41"
server_key_version: "encryptionkey@0"
I20260929 22:50:21.147559 17001 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790722213368738-14927-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790722213368738-14927-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790722213368738-14927-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 22:50:21.155835 17001 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 22:50:21.157554 17001 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 22:50:21.157672 17001 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 22:50:21.158136 17001 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 22:50:21.158174 17001 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 22:50:21.158196 17001 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 22:50:21.158208 17001 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 22:50:21.165012 17001 rpc_server.cc:307] RPC server started. Bound to: 127.14.147.195:41413
I20260929 22:50:21.165119 17131 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.147.195:41413 every 8 connection(s)
Sep 29 22:50:21 dist-test-slave-98mk krb5kdc[16603](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790722221, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.147.195@KRBTEST.COM for kudu/127.14.147.254@KRBTEST.COM
I20260929 22:50:21.165473 17001 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790722213368738-14927-0/minicluster-data/ts-2/data/info.pb
I20260929 22:50:21.168269 16716 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:50:21.158647 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.147.195:50131 (local address 127.14.147.254:35089)
0929 22:50:21.158687 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0929 22:50:21.158689 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 22:50:21.158940 (+ 251us) server_negotiation.cc:407] Connection header received
0929 22:50:21.159546 (+ 606us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 22:50:21.159547 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 22:50:21.159560 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 22:50:21.159620 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0929 22:50:21.160519 (+ 899us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:50:21.161271 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:50:21.162800 (+ 1529us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:50:21.163079 (+ 279us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:50:21.163136 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 22:50:21.165964 (+ 2828us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 22:50:21.165966 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 22:50:21.165966 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 22:50:21.165974 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0929 22:50:21.167067 (+ 1093us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 22:50:21.167512 (+ 445us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 22:50:21.167513 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 22:50:21.167513 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 22:50:21.167567 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 22:50:21.167765 (+ 198us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 22:50:21.167766 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 22:50:21.167766 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 22:50:21.167870 (+ 104us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 22:50:21.167913 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0929 22:50:21.168139 (+ 226us) server_negotiation.cc:299] Negotiation successful
0929 22:50:21.168202 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260929 22:50:21.168409 17025 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:50:21.158654 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.147.254:35089 (local address 127.14.147.195:50131)
0929 22:50:21.158814 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0929 22:50:21.158850 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0929 22:50:21.159451 (+ 601us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 22:50:21.159738 (+ 287us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 22:50:21.159744 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 22:50:21.160136 (+ 392us) client_negotiation.cc:189] Negotiated authn=SASL
0929 22:50:21.160433 (+ 297us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 22:50:21.160441 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 22:50:21.161378 (+ 937us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 22:50:21.161379 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 22:50:21.162702 (+ 1323us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 22:50:21.162706 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 22:50:21.163171 (+ 465us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 22:50:21.163172 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 22:50:21.163323 (+ 151us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 22:50:21.163995 (+ 672us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 22:50:21.164005 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0929 22:50:21.165867 (+ 1862us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 22:50:21.167153 (+ 1286us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 22:50:21.167160 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 22:50:21.167162 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 22:50:21.167405 (+ 243us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 22:50:21.167617 (+ 212us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 22:50:21.167618 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 22:50:21.167618 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 22:50:21.167716 (+ 98us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 22:50:21.167911 (+ 195us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 22:50:21.167913 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 22:50:21.168027 (+ 114us) client_negotiation.cc:769] Sending connection context
0929 22:50:21.168104 (+ 77us) client_negotiation.cc:240] Negotiation successful
0929 22:50:21.168229 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":85,"threads_started":1}
I20260929 22:50:21.168838 17132 heartbeater.cc:347] Connected to a master server at 127.14.147.254:35089
I20260929 22:50:21.168915 17132 heartbeater.cc:464] Registering TS with master...
I20260929 22:50:21.169091 17132 heartbeater.cc:511] Master 127.14.147.254:35089 requested a full tablet report, sending...
I20260929 22:50:21.169454 16648 ts_manager.cc:194] Registered new tserver with Master: 306a909ee2e344a49f9e7691e2ec8ebe (127.14.147.195:41413)
I20260929 22:50:21.169937 16648 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.147.195@KRBTEST.COM'} at 127.14.147.195:50131
I20260929 22:50:21.174743 17138 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:50:21.169767 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.147.254:35089 (local address 127.14.147.195:45307)
0929 22:50:21.169897 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0929 22:50:21.169907 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0929 22:50:21.169959 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 22:50:21.170136 (+ 177us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 22:50:21.170137 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 22:50:21.170325 (+ 188us) client_negotiation.cc:189] Negotiated authn=SASL
0929 22:50:21.170376 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 22:50:21.170378 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 22:50:21.171001 (+ 623us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 22:50:21.171002 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 22:50:21.171565 (+ 563us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 22:50:21.171567 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 22:50:21.171886 (+ 319us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 22:50:21.171887 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 22:50:21.171960 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 22:50:21.172345 (+ 385us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 22:50:21.172349 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0929 22:50:21.172617 (+ 268us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 22:50:21.173842 (+ 1225us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 22:50:21.173843 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 22:50:21.173844 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 22:50:21.173998 (+ 154us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 22:50:21.174162 (+ 164us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 22:50:21.174163 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 22:50:21.174163 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 22:50:21.174227 (+ 64us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 22:50:21.174432 (+ 205us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 22:50:21.174433 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 22:50:21.174479 (+ 46us) client_negotiation.cc:769] Sending connection context
0929 22:50:21.174538 (+ 59us) client_negotiation.cc:240] Negotiation successful
0929 22:50:21.174643 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":105,"thread_start_us":82,"threads_started":1}
I20260929 22:50:21.174742 16716 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:50:21.169834 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.147.195:45307 (local address 127.14.147.254:35089)
0929 22:50:21.169877 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0929 22:50:21.169879 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 22:50:21.169948 (+ 69us) server_negotiation.cc:407] Connection header received
0929 22:50:21.170008 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 22:50:21.170009 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 22:50:21.170017 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 22:50:21.170072 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0929 22:50:21.170448 (+ 376us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:50:21.170916 (+ 468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:50:21.171644 (+ 728us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:50:21.171833 (+ 189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:50:21.171882 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 22:50:21.172712 (+ 830us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 22:50:21.172713 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 22:50:21.172714 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 22:50:21.172720 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0929 22:50:21.173772 (+ 1052us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 22:50:21.174056 (+ 284us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 22:50:21.174057 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 22:50:21.174057 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 22:50:21.174111 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 22:50:21.174275 (+ 164us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 22:50:21.174275 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 22:50:21.174276 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 22:50:21.174381 (+ 105us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 22:50:21.174434 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0929 22:50:21.174566 (+ 132us) server_negotiation.cc:299] Negotiation successful
0929 22:50:21.174641 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260929 22:50:21.175053 14927 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhWOCLk/build/release/bin/kudu as pid 17001
I20260929 22:50:21.175158 14927 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790722213368738-14927-0/minicluster-data/ts-2/wal/instance
I20260929 22:50:21.175390 14927 external_mini_cluster.cc:1428] Setting key 564ae9aa2d00a846525b96cdd97e8d72
I20260929 22:50:21.176226 14927 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 29 22:50:21 dist-test-slave-98mk krb5kdc[16603](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790722221, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 29 22:50:21 dist-test-slave-98mk krb5kdc[16603](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790722221, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.147.254@KRBTEST.COM
I20260929 22:50:21.198738 16716 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:50:21.191844 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40032 (local address 127.14.147.254:35089)
0929 22:50:21.191951 (+ 107us) server_negotiation.cc:206] Beginning negotiation
0929 22:50:21.191954 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 22:50:21.191981 (+ 27us) server_negotiation.cc:407] Connection header received
0929 22:50:21.192100 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 22:50:21.192103 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 22:50:21.192116 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 22:50:21.192195 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0929 22:50:21.192697 (+ 502us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:50:21.193187 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:50:21.193914 (+ 727us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:50:21.194113 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:50:21.194178 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 22:50:21.196443 (+ 2265us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 22:50:21.196445 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 22:50:21.196445 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 22:50:21.196455 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0929 22:50:21.197848 (+ 1393us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 22:50:21.198183 (+ 335us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 22:50:21.198184 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 22:50:21.198185 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 22:50:21.198235 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 22:50:21.198355 (+ 120us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 22:50:21.198355 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 22:50:21.198355 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 22:50:21.198454 (+ 99us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 22:50:21.198533 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0929 22:50:21.198606 (+ 73us) server_negotiation.cc:299] Negotiation successful
0929 22:50:21.198666 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260929 22:50:21.200441 16648 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40032:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 22:50:21.202026 16648 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 22:50:21.208912 17150 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:50:21.204267 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48386 (local address 127.14.147.194:39385)
0929 22:50:21.204445 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0929 22:50:21.204449 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 22:50:21.204571 (+ 122us) server_negotiation.cc:407] Connection header received
0929 22:50:21.204682 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 22:50:21.204684 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 22:50:21.204700 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 22:50:21.204766 (+ 66us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 22:50:21.204926 (+ 160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:50:21.206061 (+ 1135us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:50:21.207674 (+ 1613us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:50:21.208416 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:50:21.208489 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 22:50:21.208531 (+ 42us) server_negotiation.cc:1035] Waiting for connection context
0929 22:50:21.208757 (+ 226us) server_negotiation.cc:299] Negotiation successful
0929 22:50:21.208804 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":140,"thread_start_us":89,"threads_started":1}
I20260929 22:50:21.208925 17146 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:50:21.204076 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.147.194:39385 (local address 127.0.0.1:48386)
0929 22:50:21.204476 (+ 400us) negotiation.cc:107] Waiting for socket to connect
0929 22:50:21.204522 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0929 22:50:21.204594 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 22:50:21.204774 (+ 180us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 22:50:21.204777 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 22:50:21.204779 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 22:50:21.204857 (+ 78us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 22:50:21.204859 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 22:50:21.206160 (+ 1301us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 22:50:21.206161 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 22:50:21.207597 (+ 1436us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 22:50:21.207598 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 22:50:21.208509 (+ 911us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 22:50:21.208510 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 22:50:21.208613 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 22:50:21.208632 (+ 19us) client_negotiation.cc:769] Sending connection context
0929 22:50:21.208731 (+ 99us) client_negotiation.cc:240] Negotiation successful
0929 22:50:21.208799 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":386,"mutex_wait_us":28,"thread_start_us":54,"threads_started":1}
I20260929 22:50:21.209515 17151 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:50:21.204371 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55110 (local address 127.14.147.193:33765)
0929 22:50:21.204533 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0929 22:50:21.204538 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 22:50:21.204666 (+ 128us) server_negotiation.cc:407] Connection header received
0929 22:50:21.204708 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 22:50:21.204710 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 22:50:21.204724 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 22:50:21.204788 (+ 64us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 22:50:21.206179 (+ 1391us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:50:21.206877 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:50:21.208015 (+ 1138us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:50:21.208557 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:50:21.208631 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 22:50:21.208684 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0929 22:50:21.209342 (+ 658us) server_negotiation.cc:299] Negotiation successful
0929 22:50:21.209393 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":124,"thread_start_us":74,"threads_started":1}
I20260929 22:50:21.208942 17147 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:50:21.204259 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.147.193:33765 (local address 127.0.0.1:55110)
0929 22:50:21.204531 (+ 272us) negotiation.cc:107] Waiting for socket to connect
0929 22:50:21.204539 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0929 22:50:21.204594 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 22:50:21.204795 (+ 201us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 22:50:21.204796 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 22:50:21.204798 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 22:50:21.204863 (+ 65us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 22:50:21.204865 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 22:50:21.206957 (+ 2092us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 22:50:21.206958 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 22:50:21.207926 (+ 968us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 22:50:21.207929 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 22:50:21.208662 (+ 733us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 22:50:21.208663 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 22:50:21.208730 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 22:50:21.208735 (+ 5us) client_negotiation.cc:769] Sending connection context
0929 22:50:21.208789 (+ 54us) client_negotiation.cc:240] Negotiation successful
0929 22:50:21.208833 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":261,"mutex_wait_us":4,"thread_start_us":34,"threads_started":1}
I20260929 22:50:21.209617 16926 tablet_service.cc:1467] Processing CreateTablet for tablet bd40e94e78f04626b23293b5dad04ea8 (DEFAULT_TABLE table=test-table [id=bedc456dc86841eba6983647aae84655]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 22:50:21.210033 17149 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:50:21.204071 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.147.195:41413 (local address 127.0.0.1:54066)
0929 22:50:21.204455 (+ 384us) negotiation.cc:107] Waiting for socket to connect
0929 22:50:21.204495 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0929 22:50:21.204594 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 22:50:21.205011 (+ 417us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 22:50:21.205012 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 22:50:21.205013 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 22:50:21.205041 (+ 28us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 22:50:21.205042 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 22:50:21.206251 (+ 1209us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 22:50:21.206252 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 22:50:21.207592 (+ 1340us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 22:50:21.207595 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 22:50:21.209783 (+ 2188us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 22:50:21.209784 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 22:50:21.209846 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 22:50:21.209850 (+ 4us) client_negotiation.cc:769] Sending connection context
0929 22:50:21.209902 (+ 52us) client_negotiation.cc:240] Negotiation successful
0929 22:50:21.209945 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":354,"mutex_wait_us":56,"thread_start_us":55,"threads_started":1}
I20260929 22:50:21.210040 16926 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bd40e94e78f04626b23293b5dad04ea8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 22:50:21.210073 17148 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:50:21.204195 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54066 (local address 127.14.147.195:41413)
0929 22:50:21.204391 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0929 22:50:21.204396 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 22:50:21.204564 (+ 168us) server_negotiation.cc:407] Connection header received
0929 22:50:21.204867 (+ 303us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 22:50:21.204869 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 22:50:21.204884 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 22:50:21.204950 (+ 66us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 22:50:21.205109 (+ 159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:50:21.206157 (+ 1048us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:50:21.208568 (+ 2411us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:50:21.209716 (+ 1148us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:50:21.209785 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 22:50:21.209842 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0929 22:50:21.209917 (+ 75us) server_negotiation.cc:299] Negotiation successful
0929 22:50:21.209957 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":161,"thread_start_us":81,"threads_started":1}
I20260929 22:50:21.210518 16786 tablet_service.cc:1467] Processing CreateTablet for tablet bd40e94e78f04626b23293b5dad04ea8 (DEFAULT_TABLE table=test-table [id=bedc456dc86841eba6983647aae84655]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 22:50:21.210793 16786 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bd40e94e78f04626b23293b5dad04ea8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 22:50:21.212955 17152 tablet_bootstrap.cc:492] T bd40e94e78f04626b23293b5dad04ea8 P cb60c60cc23349ab9e59004d1d2983fa: Bootstrap starting.
I20260929 22:50:21.213552 17153 tablet_bootstrap.cc:492] T bd40e94e78f04626b23293b5dad04ea8 P b3941f6d62734fa59b0f1005b814f025: Bootstrap starting.
I20260929 22:50:21.213608 17152 tablet_bootstrap.cc:654] T bd40e94e78f04626b23293b5dad04ea8 P cb60c60cc23349ab9e59004d1d2983fa: Neither blocks nor log segments found. Creating new log.
I20260929 22:50:21.213565 17054 tablet_service.cc:1467] Processing CreateTablet for tablet bd40e94e78f04626b23293b5dad04ea8 (DEFAULT_TABLE table=test-table [id=bedc456dc86841eba6983647aae84655]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 22:50:21.213927 17152 log.cc:824] T bd40e94e78f04626b23293b5dad04ea8 P cb60c60cc23349ab9e59004d1d2983fa: Log is configured to *not* fsync() on all Append() calls
I20260929 22:50:21.213946 17054 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bd40e94e78f04626b23293b5dad04ea8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 22:50:21.214272 17153 tablet_bootstrap.cc:654] T bd40e94e78f04626b23293b5dad04ea8 P b3941f6d62734fa59b0f1005b814f025: Neither blocks nor log segments found. Creating new log.
I20260929 22:50:21.214552 17153 log.cc:824] T bd40e94e78f04626b23293b5dad04ea8 P b3941f6d62734fa59b0f1005b814f025: Log is configured to *not* fsync() on all Append() calls
I20260929 22:50:21.214740 17152 tablet_bootstrap.cc:492] T bd40e94e78f04626b23293b5dad04ea8 P cb60c60cc23349ab9e59004d1d2983fa: No bootstrap required, opened a new log
I20260929 22:50:21.214809 17152 ts_tablet_manager.cc:1397] T bd40e94e78f04626b23293b5dad04ea8 P cb60c60cc23349ab9e59004d1d2983fa: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260929 22:50:21.215175 17153 tablet_bootstrap.cc:492] T bd40e94e78f04626b23293b5dad04ea8 P b3941f6d62734fa59b0f1005b814f025: No bootstrap required, opened a new log
I20260929 22:50:21.215234 17153 ts_tablet_manager.cc:1397] T bd40e94e78f04626b23293b5dad04ea8 P b3941f6d62734fa59b0f1005b814f025: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260929 22:50:21.216902 17153 raft_consensus.cc:348] T bd40e94e78f04626b23293b5dad04ea8 P b3941f6d62734fa59b0f1005b814f025 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb60c60cc23349ab9e59004d1d2983fa" member_type: VOTER last_known_addr { host: "127.14.147.194" port: 39385 } } peers { permanent_uuid: "306a909ee2e344a49f9e7691e2ec8ebe" member_type: VOTER last_known_addr { host: "127.14.147.195" port: 41413 } } peers { permanent_uuid: "b3941f6d62734fa59b0f1005b814f025" member_type: VOTER last_known_addr { host: "127.14.147.193" port: 33765 } }
I20260929 22:50:21.217049 17153 raft_consensus.cc:374] T bd40e94e78f04626b23293b5dad04ea8 P b3941f6d62734fa59b0f1005b814f025 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 22:50:21.217080 17153 raft_consensus.cc:729] T bd40e94e78f04626b23293b5dad04ea8 P b3941f6d62734fa59b0f1005b814f025 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b3941f6d62734fa59b0f1005b814f025, State: Initialized, Role: FOLLOWER
I20260929 22:50:21.217089 17152 raft_consensus.cc:348] T bd40e94e78f04626b23293b5dad04ea8 P cb60c60cc23349ab9e59004d1d2983fa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb60c60cc23349ab9e59004d1d2983fa" member_type: VOTER last_known_addr { host: "127.14.147.194" port: 39385 } } peers { permanent_uuid: "306a909ee2e344a49f9e7691e2ec8ebe" member_type: VOTER last_known_addr { host: "127.14.147.195" port: 41413 } } peers { permanent_uuid: "b3941f6d62734fa59b0f1005b814f025" member_type: VOTER last_known_addr { host: "127.14.147.193" port: 33765 } }
I20260929 22:50:21.217250 17152 raft_consensus.cc:374] T bd40e94e78f04626b23293b5dad04ea8 P cb60c60cc23349ab9e59004d1d2983fa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 22:50:21.217219 17153 consensus_queue.cc:260] T bd40e94e78f04626b23293b5dad04ea8 P b3941f6d62734fa59b0f1005b814f025 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb60c60cc23349ab9e59004d1d2983fa" member_type: VOTER last_known_addr { host: "127.14.147.194" port: 39385 } } peers { permanent_uuid: "306a909ee2e344a49f9e7691e2ec8ebe" member_type: VOTER last_known_addr { host: "127.14.147.195" port: 41413 } } peers { permanent_uuid: "b3941f6d62734fa59b0f1005b814f025" member_type: VOTER last_known_addr { host: "127.14.147.193" port: 33765 } }
I20260929 22:50:21.217293 17152 raft_consensus.cc:729] T bd40e94e78f04626b23293b5dad04ea8 P cb60c60cc23349ab9e59004d1d2983fa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cb60c60cc23349ab9e59004d1d2983fa, State: Initialized, Role: FOLLOWER
I20260929 22:50:21.217422 17152 consensus_queue.cc:260] T bd40e94e78f04626b23293b5dad04ea8 P cb60c60cc23349ab9e59004d1d2983fa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb60c60cc23349ab9e59004d1d2983fa" member_type: VOTER last_known_addr { host: "127.14.147.194" port: 39385 } } peers { permanent_uuid: "306a909ee2e344a49f9e7691e2ec8ebe" member_type: VOTER last_known_addr { host: "127.14.147.195" port: 41413 } } peers { permanent_uuid: "b3941f6d62734fa59b0f1005b814f025" member_type: VOTER last_known_addr { host: "127.14.147.193" port: 33765 } }
I20260929 22:50:21.217470 17153 ts_tablet_manager.cc:1428] T bd40e94e78f04626b23293b5dad04ea8 P b3941f6d62734fa59b0f1005b814f025: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260929 22:50:21.217585 16852 heartbeater.cc:503] Master 127.14.147.254:35089 was elected leader, sending a full tablet report...
I20260929 22:50:21.217680 17152 ts_tablet_manager.cc:1428] T bd40e94e78f04626b23293b5dad04ea8 P cb60c60cc23349ab9e59004d1d2983fa: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260929 22:50:21.217689 16992 heartbeater.cc:503] Master 127.14.147.254:35089 was elected leader, sending a full tablet report...
I20260929 22:50:21.217883 17156 tablet_bootstrap.cc:492] T bd40e94e78f04626b23293b5dad04ea8 P 306a909ee2e344a49f9e7691e2ec8ebe: Bootstrap starting.
I20260929 22:50:21.218884 17156 tablet_bootstrap.cc:654] T bd40e94e78f04626b23293b5dad04ea8 P 306a909ee2e344a49f9e7691e2ec8ebe: Neither blocks nor log segments found. Creating new log.
I20260929 22:50:21.219137 17156 log.cc:824] T bd40e94e78f04626b23293b5dad04ea8 P 306a909ee2e344a49f9e7691e2ec8ebe: Log is configured to *not* fsync() on all Append() calls
I20260929 22:50:21.219817 17156 tablet_bootstrap.cc:492] T bd40e94e78f04626b23293b5dad04ea8 P 306a909ee2e344a49f9e7691e2ec8ebe: No bootstrap required, opened a new log
I20260929 22:50:21.219909 17156 ts_tablet_manager.cc:1397] T bd40e94e78f04626b23293b5dad04ea8 P 306a909ee2e344a49f9e7691e2ec8ebe: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260929 22:50:21.221311 17156 raft_consensus.cc:348] T bd40e94e78f04626b23293b5dad04ea8 P 306a909ee2e344a49f9e7691e2ec8ebe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb60c60cc23349ab9e59004d1d2983fa" member_type: VOTER last_known_addr { host: "127.14.147.194" port: 39385 } } peers { permanent_uuid: "306a909ee2e344a49f9e7691e2ec8ebe" member_type: VOTER last_known_addr { host: "127.14.147.195" port: 41413 } } peers { permanent_uuid: "b3941f6d62734fa59b0f1005b814f025" member_type: VOTER last_known_addr { host: "127.14.147.193" port: 33765 } }
I20260929 22:50:21.221446 17156 raft_consensus.cc:374] T bd40e94e78f04626b23293b5dad04ea8 P 306a909ee2e344a49f9e7691e2ec8ebe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 22:50:21.221467 17156 raft_consensus.cc:729] T bd40e94e78f04626b23293b5dad04ea8 P 306a909ee2e344a49f9e7691e2ec8ebe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 306a909ee2e344a49f9e7691e2ec8ebe, State: Initialized, Role: FOLLOWER
I20260929 22:50:21.221565 17156 consensus_queue.cc:260] T bd40e94e78f04626b23293b5dad04ea8 P 306a909ee2e344a49f9e7691e2ec8ebe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb60c60cc23349ab9e59004d1d2983fa" member_type: VOTER last_known_addr { host: "127.14.147.194" port: 39385 } } peers { permanent_uuid: "306a909ee2e344a49f9e7691e2ec8ebe" member_type: VOTER last_known_addr { host: "127.14.147.195" port: 41413 } } peers { permanent_uuid: "b3941f6d62734fa59b0f1005b814f025" member_type: VOTER last_known_addr { host: "127.14.147.193" port: 33765 } }
I20260929 22:50:21.221797 17156 ts_tablet_manager.cc:1428] T bd40e94e78f04626b23293b5dad04ea8 P 306a909ee2e344a49f9e7691e2ec8ebe: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260929 22:50:21.221894 17132 heartbeater.cc:503] Master 127.14.147.254:35089 was elected leader, sending a full tablet report...
W20260929 22:50:21.272970 16993 tablet.cc:2369] T bd40e94e78f04626b23293b5dad04ea8 P cb60c60cc23349ab9e59004d1d2983fa: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 22:50:21.284744 17157 raft_consensus.cc:482] T bd40e94e78f04626b23293b5dad04ea8 P b3941f6d62734fa59b0f1005b814f025 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 22:50:21.284890 17157 raft_consensus.cc:504] T bd40e94e78f04626b23293b5dad04ea8 P b3941f6d62734fa59b0f1005b814f025 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb60c60cc23349ab9e59004d1d2983fa" member_type: VOTER last_known_addr { host: "127.14.147.194" port: 39385 } } peers { permanent_uuid: "306a909ee2e344a49f9e7691e2ec8ebe" member_type: VOTER last_known_addr { host: "127.14.147.195" port: 41413 } } peers { permanent_uuid: "b3941f6d62734fa59b0f1005b814f025" member_type: VOTER last_known_addr { host: "127.14.147.193" port: 33765 } }
I20260929 22:50:21.285246 17157 leader_election.cc:288] T bd40e94e78f04626b23293b5dad04ea8 P b3941f6d62734fa59b0f1005b814f025 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cb60c60cc23349ab9e59004d1d2983fa (127.14.147.194:39385), 306a909ee2e344a49f9e7691e2ec8ebe (127.14.147.195:41413)
I20260929 22:50:21.287899 17148 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:50:21.285517 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.147.193:36531 (local address 127.14.147.195:41413)
0929 22:50:21.285570 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0929 22:50:21.285572 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 22:50:21.285582 (+ 10us) server_negotiation.cc:407] Connection header received
0929 22:50:21.285612 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 22:50:21.285614 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 22:50:21.285622 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 22:50:21.285687 (+ 65us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 22:50:21.285838 (+ 151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:50:21.286314 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:50:21.287223 (+ 909us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:50:21.287595 (+ 372us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:50:21.287657 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 22:50:21.287667 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0929 22:50:21.287801 (+ 134us) server_negotiation.cc:299] Negotiation successful
0929 22:50:21.287835 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260929 22:50:21.287899 16745 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:50:21.285442 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.147.195:41413 (local address 127.14.147.193:36531)
0929 22:50:21.285504 (+ 62us) negotiation.cc:107] Waiting for socket to connect
0929 22:50:21.285516 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0929 22:50:21.285559 (+ 43us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 22:50:21.285704 (+ 145us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 22:50:21.285706 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 22:50:21.285707 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 22:50:21.285774 (+ 67us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 22:50:21.285776 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 22:50:21.286391 (+ 615us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 22:50:21.286392 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 22:50:21.287131 (+ 739us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 22:50:21.287133 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 22:50:21.287661 (+ 528us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 22:50:21.287662 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 22:50:21.287739 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 22:50:21.287742 (+ 3us) client_negotiation.cc:769] Sending connection context
0929 22:50:21.287785 (+ 43us) client_negotiation.cc:240] Negotiation successful
0929 22:50:21.287824 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":42}
I20260929 22:50:21.288067 17150 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:50:21.285543 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.147.193:37703 (local address 127.14.147.194:39385)
0929 22:50:21.285586 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0929 22:50:21.285588 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 22:50:21.285784 (+ 196us) server_negotiation.cc:407] Connection header received
0929 22:50:21.285828 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 22:50:21.285829 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 22:50:21.285838 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 22:50:21.285877 (+ 39us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 22:50:21.286026 (+ 149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:50:21.286497 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:50:21.287371 (+ 874us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:50:21.287769 (+ 398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:50:21.287829 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 22:50:21.287836 (+ 7us) server_negotiation.cc:1035] Waiting for connection context
0929 22:50:21.287963 (+ 127us) server_negotiation.cc:299] Negotiation successful
0929 22:50:21.288004 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260929 22:50:21.288090 17161 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:50:21.285442 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.147.194:39385 (local address 127.14.147.193:37703)
0929 22:50:21.285719 (+ 277us) negotiation.cc:107] Waiting for socket to connect
0929 22:50:21.285727 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0929 22:50:21.285772 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 22:50:21.285889 (+ 117us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 22:50:21.285890 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 22:50:21.285891 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 22:50:21.285964 (+ 73us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 22:50:21.285965 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 22:50:21.286573 (+ 608us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 22:50:21.286574 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 22:50:21.287287 (+ 713us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 22:50:21.287289 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 22:50:21.287835 (+ 546us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 22:50:21.287836 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 22:50:21.287906 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 22:50:21.287909 (+ 3us) client_negotiation.cc:769] Sending connection context
0929 22:50:21.287952 (+ 43us) client_negotiation.cc:240] Negotiation successful
0929 22:50:21.287987 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":264,"mutex_wait_us":104,"thread_start_us":108,"threads_started":1}
I20260929 22:50:21.288343 17086 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bd40e94e78f04626b23293b5dad04ea8" candidate_uuid: "b3941f6d62734fa59b0f1005b814f025" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "306a909ee2e344a49f9e7691e2ec8ebe" is_pre_election: true
I20260929 22:50:21.288509 17086 raft_consensus.cc:2454] T bd40e94e78f04626b23293b5dad04ea8 P 306a909ee2e344a49f9e7691e2ec8ebe [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b3941f6d62734fa59b0f1005b814f025 in term 0.
I20260929 22:50:21.288482 16946 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bd40e94e78f04626b23293b5dad04ea8" candidate_uuid: "b3941f6d62734fa59b0f1005b814f025" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cb60c60cc23349ab9e59004d1d2983fa" is_pre_election: true
I20260929 22:50:21.288606 16946 raft_consensus.cc:2454] T bd40e94e78f04626b23293b5dad04ea8 P cb60c60cc23349ab9e59004d1d2983fa [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b3941f6d62734fa59b0f1005b814f025 in term 0.
I20260929 22:50:21.288749 16738 leader_election.cc:302] T bd40e94e78f04626b23293b5dad04ea8 P b3941f6d62734fa59b0f1005b814f025 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 306a909ee2e344a49f9e7691e2ec8ebe, b3941f6d62734fa59b0f1005b814f025; no voters:
I20260929 22:50:21.288887 17157 raft_consensus.cc:2781] T bd40e94e78f04626b23293b5dad04ea8 P b3941f6d62734fa59b0f1005b814f025 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 22:50:21.288959 17157 raft_consensus.cc:482] T bd40e94e78f04626b23293b5dad04ea8 P b3941f6d62734fa59b0f1005b814f025 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 22:50:21.288975 17157 raft_consensus.cc:3037] T bd40e94e78f04626b23293b5dad04ea8 P b3941f6d62734fa59b0f1005b814f025 [term 0 FOLLOWER]: Advancing to term 1
I20260929 22:50:21.289674 17157 raft_consensus.cc:504] T bd40e94e78f04626b23293b5dad04ea8 P b3941f6d62734fa59b0f1005b814f025 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb60c60cc23349ab9e59004d1d2983fa" member_type: VOTER last_known_addr { host: "127.14.147.194" port: 39385 } } peers { permanent_uuid: "306a909ee2e344a49f9e7691e2ec8ebe" member_type: VOTER last_known_addr { host: "127.14.147.195" port: 41413 } } peers { permanent_uuid: "b3941f6d62734fa59b0f1005b814f025" member_type: VOTER last_known_addr { host: "127.14.147.193" port: 33765 } }
I20260929 22:50:21.289798 17157 leader_election.cc:288] T bd40e94e78f04626b23293b5dad04ea8 P b3941f6d62734fa59b0f1005b814f025 [CANDIDATE]: Term 1 election: Requested vote from peers cb60c60cc23349ab9e59004d1d2983fa (127.14.147.194:39385), 306a909ee2e344a49f9e7691e2ec8ebe (127.14.147.195:41413)
I20260929 22:50:21.289944 17086 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bd40e94e78f04626b23293b5dad04ea8" candidate_uuid: "b3941f6d62734fa59b0f1005b814f025" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "306a909ee2e344a49f9e7691e2ec8ebe"
I20260929 22:50:21.289949 16946 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bd40e94e78f04626b23293b5dad04ea8" candidate_uuid: "b3941f6d62734fa59b0f1005b814f025" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cb60c60cc23349ab9e59004d1d2983fa"
I20260929 22:50:21.290045 16946 raft_consensus.cc:3037] T bd40e94e78f04626b23293b5dad04ea8 P cb60c60cc23349ab9e59004d1d2983fa [term 0 FOLLOWER]: Advancing to term 1
I20260929 22:50:21.290045 17086 raft_consensus.cc:3037] T bd40e94e78f04626b23293b5dad04ea8 P 306a909ee2e344a49f9e7691e2ec8ebe [term 0 FOLLOWER]: Advancing to term 1
I20260929 22:50:21.290890 17086 raft_consensus.cc:2454] T bd40e94e78f04626b23293b5dad04ea8 P 306a909ee2e344a49f9e7691e2ec8ebe [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b3941f6d62734fa59b0f1005b814f025 in term 1.
I20260929 22:50:21.290890 16946 raft_consensus.cc:2454] T bd40e94e78f04626b23293b5dad04ea8 P cb60c60cc23349ab9e59004d1d2983fa [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b3941f6d62734fa59b0f1005b814f025 in term 1.
I20260929 22:50:21.291039 16738 leader_election.cc:302] T bd40e94e78f04626b23293b5dad04ea8 P b3941f6d62734fa59b0f1005b814f025 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 306a909ee2e344a49f9e7691e2ec8ebe, b3941f6d62734fa59b0f1005b814f025; no voters:
I20260929 22:50:21.291225 17157 raft_consensus.cc:2781] T bd40e94e78f04626b23293b5dad04ea8 P b3941f6d62734fa59b0f1005b814f025 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 22:50:21.291440 17157 raft_consensus.cc:686] T bd40e94e78f04626b23293b5dad04ea8 P b3941f6d62734fa59b0f1005b814f025 [term 1 LEADER]: Becoming Leader. State: Replica: b3941f6d62734fa59b0f1005b814f025, State: Running, Role: LEADER
I20260929 22:50:21.291541 17157 consensus_queue.cc:237] T bd40e94e78f04626b23293b5dad04ea8 P b3941f6d62734fa59b0f1005b814f025 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb60c60cc23349ab9e59004d1d2983fa" member_type: VOTER last_known_addr { host: "127.14.147.194" port: 39385 } } peers { permanent_uuid: "306a909ee2e344a49f9e7691e2ec8ebe" member_type: VOTER last_known_addr { host: "127.14.147.195" port: 41413 } } peers { permanent_uuid: "b3941f6d62734fa59b0f1005b814f025" member_type: VOTER last_known_addr { host: "127.14.147.193" port: 33765 } }
I20260929 22:50:21.292322 16648 catalog_manager.cc:5495] T bd40e94e78f04626b23293b5dad04ea8 P b3941f6d62734fa59b0f1005b814f025 reported cstate change: term changed from 0 to 1, leader changed from <none> to b3941f6d62734fa59b0f1005b814f025 (127.14.147.193). New cstate: current_term: 1 leader_uuid: "b3941f6d62734fa59b0f1005b814f025" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb60c60cc23349ab9e59004d1d2983fa" member_type: VOTER last_known_addr { host: "127.14.147.194" port: 39385 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "306a909ee2e344a49f9e7691e2ec8ebe" member_type: VOTER last_known_addr { host: "127.14.147.195" port: 41413 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b3941f6d62734fa59b0f1005b814f025" member_type: VOTER last_known_addr { host: "127.14.147.193" port: 33765 } health_report { overall_health: HEALTHY } } }
Sep 29 22:50:21 dist-test-slave-98mk krb5kdc[16603](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790722220, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.147.254@KRBTEST.COM for kudu/127.14.147.254@KRBTEST.COM
I20260929 22:50:21.330761 16716 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:50:21.324231 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40046 (local address 127.14.147.254:35089)
0929 22:50:21.324296 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0929 22:50:21.324299 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 22:50:21.324524 (+ 225us) server_negotiation.cc:407] Connection header received
0929 22:50:21.324577 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 22:50:21.324578 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 22:50:21.324589 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 22:50:21.324671 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0929 22:50:21.325195 (+ 524us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:50:21.325722 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:50:21.326441 (+ 719us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:50:21.326617 (+ 176us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:50:21.326671 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 22:50:21.328611 (+ 1940us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 22:50:21.328612 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 22:50:21.328613 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 22:50:21.328619 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0929 22:50:21.329739 (+ 1120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 22:50:21.330078 (+ 339us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 22:50:21.330080 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 22:50:21.330080 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 22:50:21.330136 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 22:50:21.330289 (+ 153us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 22:50:21.330290 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 22:50:21.330291 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 22:50:21.330394 (+ 103us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 22:50:21.330448 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0929 22:50:21.330578 (+ 130us) server_negotiation.cc:299] Negotiation successful
0929 22:50:21.330660 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260929 22:50:21.330834 17169 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:50:21.324184 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.147.254:35089 (local address 127.0.0.1:40046)
0929 22:50:21.324440 (+ 256us) negotiation.cc:107] Waiting for socket to connect
0929 22:50:21.324454 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 22:50:21.324531 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 22:50:21.324677 (+ 146us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 22:50:21.324677 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 22:50:21.325059 (+ 382us) client_negotiation.cc:189] Negotiated authn=SASL
0929 22:50:21.325120 (+ 61us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 22:50:21.325123 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 22:50:21.325817 (+ 694us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 22:50:21.325818 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 22:50:21.326348 (+ 530us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 22:50:21.326349 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 22:50:21.326676 (+ 327us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 22:50:21.326676 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 22:50:21.326727 (+ 51us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 22:50:21.327189 (+ 462us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 22:50:21.327193 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0929 22:50:21.328528 (+ 1335us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 22:50:21.329809 (+ 1281us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 22:50:21.329817 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 22:50:21.329820 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 22:50:21.330018 (+ 198us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 22:50:21.330188 (+ 170us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 22:50:21.330189 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 22:50:21.330189 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 22:50:21.330242 (+ 53us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 22:50:21.330444 (+ 202us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 22:50:21.330451 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 22:50:21.330496 (+ 45us) client_negotiation.cc:769] Sending connection context
0929 22:50:21.330553 (+ 57us) client_negotiation.cc:240] Negotiation successful
0929 22:50:21.330729 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":219,"mutex_wait_us":69,"thread_start_us":98,"threads_started":1}
I20260929 22:50:21.333195 16648 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.14.147.254@KRBTEST.COM'} at 127.0.0.1:40046:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260929 22:50:21.333439 16648 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 22:50:21.334880 17054 tablet_service.cc:1467] Processing CreateTablet for tablet 29bf225079494742b8c7bb0cbec81cf0 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=a9c7116bd01c45b2a4ca7b2b1149f136]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260929 22:50:21.334877 16926 tablet_service.cc:1467] Processing CreateTablet for tablet 29bf225079494742b8c7bb0cbec81cf0 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=a9c7116bd01c45b2a4ca7b2b1149f136]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260929 22:50:21.335029 16926 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 29bf225079494742b8c7bb0cbec81cf0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 22:50:21.335049 17054 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 29bf225079494742b8c7bb0cbec81cf0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 22:50:21.336344 17156 tablet_bootstrap.cc:492] T 29bf225079494742b8c7bb0cbec81cf0 P 306a909ee2e344a49f9e7691e2ec8ebe: Bootstrap starting.
I20260929 22:50:21.336453 16786 tablet_service.cc:1467] Processing CreateTablet for tablet 29bf225079494742b8c7bb0cbec81cf0 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=a9c7116bd01c45b2a4ca7b2b1149f136]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260929 22:50:21.336617 16786 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 29bf225079494742b8c7bb0cbec81cf0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 22:50:21.336882 17156 tablet_bootstrap.cc:654] T 29bf225079494742b8c7bb0cbec81cf0 P 306a909ee2e344a49f9e7691e2ec8ebe: Neither blocks nor log segments found. Creating new log.
I20260929 22:50:21.337203 17152 tablet_bootstrap.cc:492] T 29bf225079494742b8c7bb0cbec81cf0 P cb60c60cc23349ab9e59004d1d2983fa: Bootstrap starting.
I20260929 22:50:21.337731 17152 tablet_bootstrap.cc:654] T 29bf225079494742b8c7bb0cbec81cf0 P cb60c60cc23349ab9e59004d1d2983fa: Neither blocks nor log segments found. Creating new log.
I20260929 22:50:21.337949 17153 tablet_bootstrap.cc:492] T 29bf225079494742b8c7bb0cbec81cf0 P b3941f6d62734fa59b0f1005b814f025: Bootstrap starting.
I20260929 22:50:21.338179 17156 tablet_bootstrap.cc:492] T 29bf225079494742b8c7bb0cbec81cf0 P 306a909ee2e344a49f9e7691e2ec8ebe: No bootstrap required, opened a new log
I20260929 22:50:21.338230 17156 ts_tablet_manager.cc:1397] T 29bf225079494742b8c7bb0cbec81cf0 P 306a909ee2e344a49f9e7691e2ec8ebe: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260929 22:50:21.338359 17153 tablet_bootstrap.cc:654] T 29bf225079494742b8c7bb0cbec81cf0 P b3941f6d62734fa59b0f1005b814f025: Neither blocks nor log segments found. Creating new log.
I20260929 22:50:21.338389 17156 raft_consensus.cc:348] T 29bf225079494742b8c7bb0cbec81cf0 P 306a909ee2e344a49f9e7691e2ec8ebe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3941f6d62734fa59b0f1005b814f025" member_type: VOTER last_known_addr { host: "127.14.147.193" port: 33765 } } peers { permanent_uuid: "cb60c60cc23349ab9e59004d1d2983fa" member_type: VOTER last_known_addr { host: "127.14.147.194" port: 39385 } } peers { permanent_uuid: "306a909ee2e344a49f9e7691e2ec8ebe" member_type: VOTER last_known_addr { host: "127.14.147.195" port: 41413 } }
I20260929 22:50:21.338456 17152 tablet_bootstrap.cc:492] T 29bf225079494742b8c7bb0cbec81cf0 P cb60c60cc23349ab9e59004d1d2983fa: No bootstrap required, opened a new log
I20260929 22:50:21.338474 17156 raft_consensus.cc:374] T 29bf225079494742b8c7bb0cbec81cf0 P 306a909ee2e344a49f9e7691e2ec8ebe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 22:50:21.338501 17156 raft_consensus.cc:729] T 29bf225079494742b8c7bb0cbec81cf0 P 306a909ee2e344a49f9e7691e2ec8ebe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 306a909ee2e344a49f9e7691e2ec8ebe, State: Initialized, Role: FOLLOWER
I20260929 22:50:21.338500 17152 ts_tablet_manager.cc:1397] T 29bf225079494742b8c7bb0cbec81cf0 P cb60c60cc23349ab9e59004d1d2983fa: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260929 22:50:21.338546 17156 consensus_queue.cc:260] T 29bf225079494742b8c7bb0cbec81cf0 P 306a909ee2e344a49f9e7691e2ec8ebe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3941f6d62734fa59b0f1005b814f025" member_type: VOTER last_known_addr { host: "127.14.147.193" port: 33765 } } peers { permanent_uuid: "cb60c60cc23349ab9e59004d1d2983fa" member_type: VOTER last_known_addr { host: "127.14.147.194" port: 39385 } } peers { permanent_uuid: "306a909ee2e344a49f9e7691e2ec8ebe" member_type: VOTER last_known_addr { host: "127.14.147.195" port: 41413 } }
I20260929 22:50:21.338634 17156 ts_tablet_manager.cc:1428] T 29bf225079494742b8c7bb0cbec81cf0 P 306a909ee2e344a49f9e7691e2ec8ebe: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260929 22:50:21.338653 17152 raft_consensus.cc:348] T 29bf225079494742b8c7bb0cbec81cf0 P cb60c60cc23349ab9e59004d1d2983fa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3941f6d62734fa59b0f1005b814f025" member_type: VOTER last_known_addr { host: "127.14.147.193" port: 33765 } } peers { permanent_uuid: "cb60c60cc23349ab9e59004d1d2983fa" member_type: VOTER last_known_addr { host: "127.14.147.194" port: 39385 } } peers { permanent_uuid: "306a909ee2e344a49f9e7691e2ec8ebe" member_type: VOTER last_known_addr { host: "127.14.147.195" port: 41413 } }
I20260929 22:50:21.338733 17152 raft_consensus.cc:374] T 29bf225079494742b8c7bb0cbec81cf0 P cb60c60cc23349ab9e59004d1d2983fa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 22:50:21.338758 17152 raft_consensus.cc:729] T 29bf225079494742b8c7bb0cbec81cf0 P cb60c60cc23349ab9e59004d1d2983fa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cb60c60cc23349ab9e59004d1d2983fa, State: Initialized, Role: FOLLOWER
I20260929 22:50:21.338798 17152 consensus_queue.cc:260] T 29bf225079494742b8c7bb0cbec81cf0 P cb60c60cc23349ab9e59004d1d2983fa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3941f6d62734fa59b0f1005b814f025" member_type: VOTER last_known_addr { host: "127.14.147.193" port: 33765 } } peers { permanent_uuid: "cb60c60cc23349ab9e59004d1d2983fa" member_type: VOTER last_known_addr { host: "127.14.147.194" port: 39385 } } peers { permanent_uuid: "306a909ee2e344a49f9e7691e2ec8ebe" member_type: VOTER last_known_addr { host: "127.14.147.195" port: 41413 } }
I20260929 22:50:21.338781 17160 tablet_replica.cc:439] T 29bf225079494742b8c7bb0cbec81cf0 P 306a909ee2e344a49f9e7691e2ec8ebe: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3941f6d62734fa59b0f1005b814f025" member_type: VOTER last_known_addr { host: "127.14.147.193" port: 33765 } } peers { permanent_uuid: "cb60c60cc23349ab9e59004d1d2983fa" member_type: VOTER last_known_addr { host: "127.14.147.194" port: 39385 } } peers { permanent_uuid: "306a909ee2e344a49f9e7691e2ec8ebe" member_type: VOTER last_known_addr { host: "127.14.147.195" port: 41413 } } }
I20260929 22:50:21.338855 17160 tablet_replica.cc:442] T 29bf225079494742b8c7bb0cbec81cf0 P 306a909ee2e344a49f9e7691e2ec8ebe: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 22:50:21.338883 17152 ts_tablet_manager.cc:1428] T 29bf225079494742b8c7bb0cbec81cf0 P cb60c60cc23349ab9e59004d1d2983fa: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260929 22:50:21.338969 17153 tablet_bootstrap.cc:492] T 29bf225079494742b8c7bb0cbec81cf0 P b3941f6d62734fa59b0f1005b814f025: No bootstrap required, opened a new log
I20260929 22:50:21.339006 17153 ts_tablet_manager.cc:1397] T 29bf225079494742b8c7bb0cbec81cf0 P b3941f6d62734fa59b0f1005b814f025: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260929 22:50:21.339143 17153 raft_consensus.cc:348] T 29bf225079494742b8c7bb0cbec81cf0 P b3941f6d62734fa59b0f1005b814f025 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3941f6d62734fa59b0f1005b814f025" member_type: VOTER last_known_addr { host: "127.14.147.193" port: 33765 } } peers { permanent_uuid: "cb60c60cc23349ab9e59004d1d2983fa" member_type: VOTER last_known_addr { host: "127.14.147.194" port: 39385 } } peers { permanent_uuid: "306a909ee2e344a49f9e7691e2ec8ebe" member_type: VOTER last_known_addr { host: "127.14.147.195" port: 41413 } }
I20260929 22:50:21.339217 17153 raft_consensus.cc:374] T 29bf225079494742b8c7bb0cbec81cf0 P b3941f6d62734fa59b0f1005b814f025 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 22:50:21.339236 17153 raft_consensus.cc:729] T 29bf225079494742b8c7bb0cbec81cf0 P b3941f6d62734fa59b0f1005b814f025 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b3941f6d62734fa59b0f1005b814f025, State: Initialized, Role: FOLLOWER
I20260929 22:50:21.339277 17153 consensus_queue.cc:260] T 29bf225079494742b8c7bb0cbec81cf0 P b3941f6d62734fa59b0f1005b814f025 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3941f6d62734fa59b0f1005b814f025" member_type: VOTER last_known_addr { host: "127.14.147.193" port: 33765 } } peers { permanent_uuid: "cb60c60cc23349ab9e59004d1d2983fa" member_type: VOTER last_known_addr { host: "127.14.147.194" port: 39385 } } peers { permanent_uuid: "306a909ee2e344a49f9e7691e2ec8ebe" member_type: VOTER last_known_addr { host: "127.14.147.195" port: 41413 } }
I20260929 22:50:21.339361 17153 ts_tablet_manager.cc:1428] T 29bf225079494742b8c7bb0cbec81cf0 P b3941f6d62734fa59b0f1005b814f025: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260929 22:50:21.339411 17157 tablet_replica.cc:439] T 29bf225079494742b8c7bb0cbec81cf0 P b3941f6d62734fa59b0f1005b814f025: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3941f6d62734fa59b0f1005b814f025" member_type: VOTER last_known_addr { host: "127.14.147.193" port: 33765 } } peers { permanent_uuid: "cb60c60cc23349ab9e59004d1d2983fa" member_type: VOTER last_known_addr { host: "127.14.147.194" port: 39385 } } peers { permanent_uuid: "306a909ee2e344a49f9e7691e2ec8ebe" member_type: VOTER last_known_addr { host: "127.14.147.195" port: 41413 } } }
I20260929 22:50:21.339529 17157 tablet_replica.cc:442] T 29bf225079494742b8c7bb0cbec81cf0 P b3941f6d62734fa59b0f1005b814f025: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 22:50:21.339548 17158 tablet_replica.cc:439] T 29bf225079494742b8c7bb0cbec81cf0 P cb60c60cc23349ab9e59004d1d2983fa: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3941f6d62734fa59b0f1005b814f025" member_type: VOTER last_known_addr { host: "127.14.147.193" port: 33765 } } peers { permanent_uuid: "cb60c60cc23349ab9e59004d1d2983fa" member_type: VOTER last_known_addr { host: "127.14.147.194" port: 39385 } } peers { permanent_uuid: "306a909ee2e344a49f9e7691e2ec8ebe" member_type: VOTER last_known_addr { host: "127.14.147.195" port: 41413 } } }
I20260929 22:50:21.339643 17158 tablet_replica.cc:442] T 29bf225079494742b8c7bb0cbec81cf0 P cb60c60cc23349ab9e59004d1d2983fa: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 22:50:21.346518 17157 raft_consensus.cc:482] T 29bf225079494742b8c7bb0cbec81cf0 P b3941f6d62734fa59b0f1005b814f025 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 22:50:21.346611 17157 raft_consensus.cc:504] T 29bf225079494742b8c7bb0cbec81cf0 P b3941f6d62734fa59b0f1005b814f025 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3941f6d62734fa59b0f1005b814f025" member_type: VOTER last_known_addr { host: "127.14.147.193" port: 33765 } } peers { permanent_uuid: "cb60c60cc23349ab9e59004d1d2983fa" member_type: VOTER last_known_addr { host: "127.14.147.194" port: 39385 } } peers { permanent_uuid: "306a909ee2e344a49f9e7691e2ec8ebe" member_type: VOTER last_known_addr { host: "127.14.147.195" port: 41413 } }
I20260929 22:50:21.346742 17157 leader_election.cc:288] T 29bf225079494742b8c7bb0cbec81cf0 P b3941f6d62734fa59b0f1005b814f025 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cb60c60cc23349ab9e59004d1d2983fa (127.14.147.194:39385), 306a909ee2e344a49f9e7691e2ec8ebe (127.14.147.195:41413)
I20260929 22:50:21.346933 16946 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "29bf225079494742b8c7bb0cbec81cf0" candidate_uuid: "b3941f6d62734fa59b0f1005b814f025" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cb60c60cc23349ab9e59004d1d2983fa" is_pre_election: true
I20260929 22:50:21.346985 17086 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "29bf225079494742b8c7bb0cbec81cf0" candidate_uuid: "b3941f6d62734fa59b0f1005b814f025" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "306a909ee2e344a49f9e7691e2ec8ebe" is_pre_election: true
I20260929 22:50:21.347061 17086 raft_consensus.cc:2454] T 29bf225079494742b8c7bb0cbec81cf0 P 306a909ee2e344a49f9e7691e2ec8ebe [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b3941f6d62734fa59b0f1005b814f025 in term 0.
I20260929 22:50:21.347055 16946 raft_consensus.cc:2454] T 29bf225079494742b8c7bb0cbec81cf0 P cb60c60cc23349ab9e59004d1d2983fa [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b3941f6d62734fa59b0f1005b814f025 in term 0.
I20260929 22:50:21.347200 16738 leader_election.cc:302] T 29bf225079494742b8c7bb0cbec81cf0 P b3941f6d62734fa59b0f1005b814f025 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 306a909ee2e344a49f9e7691e2ec8ebe, b3941f6d62734fa59b0f1005b814f025; no voters:
I20260929 22:50:21.347321 17157 raft_consensus.cc:2781] T 29bf225079494742b8c7bb0cbec81cf0 P b3941f6d62734fa59b0f1005b814f025 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 22:50:21.347371 17157 raft_consensus.cc:482] T 29bf225079494742b8c7bb0cbec81cf0 P b3941f6d62734fa59b0f1005b814f025 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 22:50:21.347390 17157 raft_consensus.cc:3037] T 29bf225079494742b8c7bb0cbec81cf0 P b3941f6d62734fa59b0f1005b814f025 [term 0 FOLLOWER]: Advancing to term 1
I20260929 22:50:21.348084 17157 raft_consensus.cc:504] T 29bf225079494742b8c7bb0cbec81cf0 P b3941f6d62734fa59b0f1005b814f025 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3941f6d62734fa59b0f1005b814f025" member_type: VOTER last_known_addr { host: "127.14.147.193" port: 33765 } } peers { permanent_uuid: "cb60c60cc23349ab9e59004d1d2983fa" member_type: VOTER last_known_addr { host: "127.14.147.194" port: 39385 } } peers { permanent_uuid: "306a909ee2e344a49f9e7691e2ec8ebe" member_type: VOTER last_known_addr { host: "127.14.147.195" port: 41413 } }
I20260929 22:50:21.348203 17157 leader_election.cc:288] T 29bf225079494742b8c7bb0cbec81cf0 P b3941f6d62734fa59b0f1005b814f025 [CANDIDATE]: Term 1 election: Requested vote from peers cb60c60cc23349ab9e59004d1d2983fa (127.14.147.194:39385), 306a909ee2e344a49f9e7691e2ec8ebe (127.14.147.195:41413)
I20260929 22:50:21.348372 17086 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "29bf225079494742b8c7bb0cbec81cf0" candidate_uuid: "b3941f6d62734fa59b0f1005b814f025" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "306a909ee2e344a49f9e7691e2ec8ebe"
I20260929 22:50:21.348381 16946 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "29bf225079494742b8c7bb0cbec81cf0" candidate_uuid: "b3941f6d62734fa59b0f1005b814f025" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cb60c60cc23349ab9e59004d1d2983fa"
I20260929 22:50:21.348446 16946 raft_consensus.cc:3037] T 29bf225079494742b8c7bb0cbec81cf0 P cb60c60cc23349ab9e59004d1d2983fa [term 0 FOLLOWER]: Advancing to term 1
I20260929 22:50:21.348452 17086 raft_consensus.cc:3037] T 29bf225079494742b8c7bb0cbec81cf0 P 306a909ee2e344a49f9e7691e2ec8ebe [term 0 FOLLOWER]: Advancing to term 1
I20260929 22:50:21.349138 17086 raft_consensus.cc:2454] T 29bf225079494742b8c7bb0cbec81cf0 P 306a909ee2e344a49f9e7691e2ec8ebe [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b3941f6d62734fa59b0f1005b814f025 in term 1.
I20260929 22:50:21.349155 16946 raft_consensus.cc:2454] T 29bf225079494742b8c7bb0cbec81cf0 P cb60c60cc23349ab9e59004d1d2983fa [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b3941f6d62734fa59b0f1005b814f025 in term 1.
I20260929 22:50:21.349318 16740 leader_election.cc:302] T 29bf225079494742b8c7bb0cbec81cf0 P b3941f6d62734fa59b0f1005b814f025 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b3941f6d62734fa59b0f1005b814f025, cb60c60cc23349ab9e59004d1d2983fa; no voters:
I20260929 22:50:21.349416 17157 raft_consensus.cc:2781] T 29bf225079494742b8c7bb0cbec81cf0 P b3941f6d62734fa59b0f1005b814f025 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 22:50:21.349476 17157 raft_consensus.cc:686] T 29bf225079494742b8c7bb0cbec81cf0 P b3941f6d62734fa59b0f1005b814f025 [term 1 LEADER]: Becoming Leader. State: Replica: b3941f6d62734fa59b0f1005b814f025, State: Running, Role: LEADER
I20260929 22:50:21.349536 17157 consensus_queue.cc:237] T 29bf225079494742b8c7bb0cbec81cf0 P b3941f6d62734fa59b0f1005b814f025 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3941f6d62734fa59b0f1005b814f025" member_type: VOTER last_known_addr { host: "127.14.147.193" port: 33765 } } peers { permanent_uuid: "cb60c60cc23349ab9e59004d1d2983fa" member_type: VOTER last_known_addr { host: "127.14.147.194" port: 39385 } } peers { permanent_uuid: "306a909ee2e344a49f9e7691e2ec8ebe" member_type: VOTER last_known_addr { host: "127.14.147.195" port: 41413 } }
I20260929 22:50:21.349830 17162 tablet_replica.cc:439] T 29bf225079494742b8c7bb0cbec81cf0 P b3941f6d62734fa59b0f1005b814f025: TxnStatusTablet state changed. Reason: New leader b3941f6d62734fa59b0f1005b814f025. Latest consensus state: current_term: 1 leader_uuid: "b3941f6d62734fa59b0f1005b814f025" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3941f6d62734fa59b0f1005b814f025" member_type: VOTER last_known_addr { host: "127.14.147.193" port: 33765 } } peers { permanent_uuid: "cb60c60cc23349ab9e59004d1d2983fa" member_type: VOTER last_known_addr { host: "127.14.147.194" port: 39385 } } peers { permanent_uuid: "306a909ee2e344a49f9e7691e2ec8ebe" member_type: VOTER last_known_addr { host: "127.14.147.195" port: 41413 } } }
I20260929 22:50:21.349898 17162 tablet_replica.cc:442] T 29bf225079494742b8c7bb0cbec81cf0 P b3941f6d62734fa59b0f1005b814f025: This TxnStatusTablet replica's current role is: LEADER
I20260929 22:50:21.350143 17174 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260929 22:50:21.350165 16646 catalog_manager.cc:5495] T 29bf225079494742b8c7bb0cbec81cf0 P b3941f6d62734fa59b0f1005b814f025 reported cstate change: term changed from 0 to 1, leader changed from <none> to b3941f6d62734fa59b0f1005b814f025 (127.14.147.193). New cstate: current_term: 1 leader_uuid: "b3941f6d62734fa59b0f1005b814f025" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3941f6d62734fa59b0f1005b814f025" member_type: VOTER last_known_addr { host: "127.14.147.193" port: 33765 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cb60c60cc23349ab9e59004d1d2983fa" member_type: VOTER last_known_addr { host: "127.14.147.194" port: 39385 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "306a909ee2e344a49f9e7691e2ec8ebe" member_type: VOTER last_known_addr { host: "127.14.147.195" port: 41413 } health_report { overall_health: UNKNOWN } } }
I20260929 22:50:21.350275 17174 txn_status_manager.cc:929] Loading transaction status metadata into memory...
W20260929 22:50:21.357111 16853 tablet.cc:2369] T 29bf225079494742b8c7bb0cbec81cf0 P b3941f6d62734fa59b0f1005b814f025: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 22:50:21.357860 17169 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:50:21.354884 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.147.193:33765 (local address 127.0.0.1:55126)
0929 22:50:21.354970 (+ 86us) negotiation.cc:107] Waiting for socket to connect
0929 22:50:21.354982 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0929 22:50:21.355039 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 22:50:21.355200 (+ 161us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 22:50:21.355200 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 22:50:21.355202 (+ 2us) client_negotiation.cc:189] Negotiated authn=TOKEN
0929 22:50:21.355258 (+ 56us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 22:50:21.355261 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 22:50:21.355953 (+ 692us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 22:50:21.355954 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 22:50:21.356872 (+ 918us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 22:50:21.356875 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 22:50:21.357358 (+ 483us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 22:50:21.357359 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 22:50:21.357425 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 22:50:21.357442 (+ 17us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0929 22:50:21.357730 (+ 288us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0929 22:50:21.357734 (+ 4us) client_negotiation.cc:769] Sending connection context
0929 22:50:21.357757 (+ 23us) client_negotiation.cc:240] Negotiation successful
0929 22:50:21.357800 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":50}
I20260929 22:50:21.357868 17151 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:50:21.354972 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55126 (local address 127.14.147.193:33765)
0929 22:50:21.355013 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0929 22:50:21.355015 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 22:50:21.355025 (+ 10us) server_negotiation.cc:407] Connection header received
0929 22:50:21.355111 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 22:50:21.355112 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 22:50:21.355129 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 22:50:21.355195 (+ 66us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 22:50:21.355315 (+ 120us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:50:21.355876 (+ 561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:50:21.356963 (+ 1087us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:50:21.357251 (+ 288us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:50:21.357312 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 22:50:21.357535 (+ 223us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 22:50:21.357663 (+ 128us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 22:50:21.357752 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0929 22:50:21.357769 (+ 17us) server_negotiation.cc:299] Negotiation successful
0929 22:50:21.357794 (+ 25us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260929 22:50:21.359862 16946 raft_consensus.cc:1261] T 29bf225079494742b8c7bb0cbec81cf0 P cb60c60cc23349ab9e59004d1d2983fa [term 1 FOLLOWER]: Refusing update from remote peer b3941f6d62734fa59b0f1005b814f025: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 22:50:21.359931 17086 raft_consensus.cc:1261] T 29bf225079494742b8c7bb0cbec81cf0 P 306a909ee2e344a49f9e7691e2ec8ebe [term 1 FOLLOWER]: Refusing update from remote peer b3941f6d62734fa59b0f1005b814f025: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 22:50:21.360298 17162 consensus_queue.cc:1035] T 29bf225079494742b8c7bb0cbec81cf0 P b3941f6d62734fa59b0f1005b814f025 [LEADER]: Connected to new peer: Peer: permanent_uuid: "306a909ee2e344a49f9e7691e2ec8ebe" member_type: VOTER last_known_addr { host: "127.14.147.195" port: 41413 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 22:50:21.360427 17157 consensus_queue.cc:1035] T 29bf225079494742b8c7bb0cbec81cf0 P b3941f6d62734fa59b0f1005b814f025 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cb60c60cc23349ab9e59004d1d2983fa" member_type: VOTER last_known_addr { host: "127.14.147.194" port: 39385 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 22:50:21.361624 17160 tablet_replica.cc:439] T 29bf225079494742b8c7bb0cbec81cf0 P 306a909ee2e344a49f9e7691e2ec8ebe: TxnStatusTablet state changed. Reason: New leader b3941f6d62734fa59b0f1005b814f025. Latest consensus state: current_term: 1 leader_uuid: "b3941f6d62734fa59b0f1005b814f025" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3941f6d62734fa59b0f1005b814f025" member_type: VOTER last_known_addr { host: "127.14.147.193" port: 33765 } } peers { permanent_uuid: "cb60c60cc23349ab9e59004d1d2983fa" member_type: VOTER last_known_addr { host: "127.14.147.194" port: 39385 } } peers { permanent_uuid: "306a909ee2e344a49f9e7691e2ec8ebe" member_type: VOTER last_known_addr { host: "127.14.147.195" port: 41413 } } }
I20260929 22:50:21.361671 17158 tablet_replica.cc:439] T 29bf225079494742b8c7bb0cbec81cf0 P cb60c60cc23349ab9e59004d1d2983fa: TxnStatusTablet state changed. Reason: New leader b3941f6d62734fa59b0f1005b814f025. Latest consensus state: current_term: 1 leader_uuid: "b3941f6d62734fa59b0f1005b814f025" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3941f6d62734fa59b0f1005b814f025" member_type: VOTER last_known_addr { host: "127.14.147.193" port: 33765 } } peers { permanent_uuid: "cb60c60cc23349ab9e59004d1d2983fa" member_type: VOTER last_known_addr { host: "127.14.147.194" port: 39385 } } peers { permanent_uuid: "306a909ee2e344a49f9e7691e2ec8ebe" member_type: VOTER last_known_addr { host: "127.14.147.195" port: 41413 } } }
I20260929 22:50:21.361752 17160 tablet_replica.cc:442] T 29bf225079494742b8c7bb0cbec81cf0 P 306a909ee2e344a49f9e7691e2ec8ebe: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 22:50:21.361770 17158 tablet_replica.cc:442] T 29bf225079494742b8c7bb0cbec81cf0 P cb60c60cc23349ab9e59004d1d2983fa: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 22:50:21.362762 17157 tablet_replica.cc:439] T 29bf225079494742b8c7bb0cbec81cf0 P b3941f6d62734fa59b0f1005b814f025: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "b3941f6d62734fa59b0f1005b814f025" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3941f6d62734fa59b0f1005b814f025" member_type: VOTER last_known_addr { host: "127.14.147.193" port: 33765 } } peers { permanent_uuid: "cb60c60cc23349ab9e59004d1d2983fa" member_type: VOTER last_known_addr { host: "127.14.147.194" port: 39385 } } peers { permanent_uuid: "306a909ee2e344a49f9e7691e2ec8ebe" member_type: VOTER last_known_addr { host: "127.14.147.195" port: 41413 } } }
I20260929 22:50:21.362854 17157 tablet_replica.cc:442] T 29bf225079494742b8c7bb0cbec81cf0 P b3941f6d62734fa59b0f1005b814f025: This TxnStatusTablet replica's current role is: LEADER
I20260929 22:50:21.362893 17157 tablet_replica.cc:439] T 29bf225079494742b8c7bb0cbec81cf0 P b3941f6d62734fa59b0f1005b814f025: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "b3941f6d62734fa59b0f1005b814f025" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3941f6d62734fa59b0f1005b814f025" member_type: VOTER last_known_addr { host: "127.14.147.193" port: 33765 } } peers { permanent_uuid: "cb60c60cc23349ab9e59004d1d2983fa" member_type: VOTER last_known_addr { host: "127.14.147.194" port: 39385 } } peers { permanent_uuid: "306a909ee2e344a49f9e7691e2ec8ebe" member_type: VOTER last_known_addr { host: "127.14.147.195" port: 41413 } } }
I20260929 22:50:21.362936 17157 tablet_replica.cc:442] T 29bf225079494742b8c7bb0cbec81cf0 P b3941f6d62734fa59b0f1005b814f025: This TxnStatusTablet replica's current role is: LEADER
I20260929 22:50:21.363263 17178 mvcc.cc:205] Tried to move back new op lower bound from 7334798218687344640 to 7334798218648154112. Current Snapshot: MvccSnapshot[applied={T|T < 7334798218687344640}]
I20260929 22:50:21.363266 17158 tablet_replica.cc:439] T 29bf225079494742b8c7bb0cbec81cf0 P cb60c60cc23349ab9e59004d1d2983fa: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "b3941f6d62734fa59b0f1005b814f025" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3941f6d62734fa59b0f1005b814f025" member_type: VOTER last_known_addr { host: "127.14.147.193" port: 33765 } } peers { permanent_uuid: "cb60c60cc23349ab9e59004d1d2983fa" member_type: VOTER last_known_addr { host: "127.14.147.194" port: 39385 } } peers { permanent_uuid: "306a909ee2e344a49f9e7691e2ec8ebe" member_type: VOTER last_known_addr { host: "127.14.147.195" port: 41413 } } }
I20260929 22:50:21.363349 17158 tablet_replica.cc:442] T 29bf225079494742b8c7bb0cbec81cf0 P cb60c60cc23349ab9e59004d1d2983fa: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 22:50:21.363500 17160 tablet_replica.cc:439] T 29bf225079494742b8c7bb0cbec81cf0 P 306a909ee2e344a49f9e7691e2ec8ebe: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "b3941f6d62734fa59b0f1005b814f025" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3941f6d62734fa59b0f1005b814f025" member_type: VOTER last_known_addr { host: "127.14.147.193" port: 33765 } } peers { permanent_uuid: "cb60c60cc23349ab9e59004d1d2983fa" member_type: VOTER last_known_addr { host: "127.14.147.194" port: 39385 } } peers { permanent_uuid: "306a909ee2e344a49f9e7691e2ec8ebe" member_type: VOTER last_known_addr { host: "127.14.147.195" port: 41413 } } }
I20260929 22:50:21.363577 17160 tablet_replica.cc:442] T 29bf225079494742b8c7bb0cbec81cf0 P 306a909ee2e344a49f9e7691e2ec8ebe: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 22:50:21.364289 17179 mvcc.cc:205] Tried to move back new op lower bound from 7334798218687344640 to 7334798218648154112. Current Snapshot: MvccSnapshot[applied={T|T < 7334798218687344640}]
I20260929 22:50:21.369093 17151 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:50:21.366056 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55132 (local address 127.14.147.193:33765)
0929 22:50:21.366112 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0929 22:50:21.366114 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 22:50:21.366122 (+ 8us) server_negotiation.cc:407] Connection header received
0929 22:50:21.366143 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 22:50:21.366144 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 22:50:21.366153 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 22:50:21.366208 (+ 55us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 22:50:21.366388 (+ 180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:50:21.367139 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:50:21.368029 (+ 890us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:50:21.368344 (+ 315us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:50:21.368418 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 22:50:21.368770 (+ 352us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 22:50:21.368819 (+ 49us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 22:50:21.368862 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0929 22:50:21.368957 (+ 95us) server_negotiation.cc:299] Negotiation successful
0929 22:50:21.368995 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260929 22:50:21.373898 17151 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:50:21.371600 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.147.193:36393 (local address 127.14.147.193:33765)
0929 22:50:21.371640 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0929 22:50:21.371642 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 22:50:21.371783 (+ 141us) server_negotiation.cc:407] Connection header received
0929 22:50:21.371845 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 22:50:21.371846 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 22:50:21.371855 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 22:50:21.371878 (+ 23us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 22:50:21.372062 (+ 184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:50:21.372520 (+ 458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:50:21.373232 (+ 712us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:50:21.373421 (+ 189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:50:21.373473 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 22:50:21.373636 (+ 163us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 22:50:21.373672 (+ 36us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 22:50:21.373715 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0929 22:50:21.373769 (+ 54us) server_negotiation.cc:299] Negotiation successful
0929 22:50:21.373786 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260929 22:50:21.373911 17188 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:50:21.371548 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.147.193:33765 (local address 127.14.147.193:36393)
0929 22:50:21.371722 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0929 22:50:21.371731 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0929 22:50:21.371783 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 22:50:21.371933 (+ 150us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 22:50:21.371933 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 22:50:21.371934 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0929 22:50:21.372008 (+ 74us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 22:50:21.372009 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 22:50:21.372587 (+ 578us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 22:50:21.372588 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 22:50:21.373161 (+ 573us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 22:50:21.373162 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 22:50:21.373475 (+ 313us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 22:50:21.373476 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 22:50:21.373539 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 22:50:21.373556 (+ 17us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0929 22:50:21.373720 (+ 164us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0929 22:50:21.373721 (+ 1us) client_negotiation.cc:769] Sending connection context
0929 22:50:21.373758 (+ 37us) client_negotiation.cc:240] Negotiation successful
0929 22:50:21.373801 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":145,"thread_start_us":83,"threads_started":1}
I20260929 22:50:21.376134 16946 raft_consensus.cc:1261] T bd40e94e78f04626b23293b5dad04ea8 P cb60c60cc23349ab9e59004d1d2983fa [term 1 FOLLOWER]: Refusing update from remote peer b3941f6d62734fa59b0f1005b814f025: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 22:50:21.376174 17086 raft_consensus.cc:1261] T bd40e94e78f04626b23293b5dad04ea8 P 306a909ee2e344a49f9e7691e2ec8ebe [term 1 FOLLOWER]: Refusing update from remote peer b3941f6d62734fa59b0f1005b814f025: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 22:50:21.376358 17162 consensus_queue.cc:1035] T bd40e94e78f04626b23293b5dad04ea8 P b3941f6d62734fa59b0f1005b814f025 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cb60c60cc23349ab9e59004d1d2983fa" member_type: VOTER last_known_addr { host: "127.14.147.194" port: 39385 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 22:50:21.376451 17157 consensus_queue.cc:1035] T bd40e94e78f04626b23293b5dad04ea8 P b3941f6d62734fa59b0f1005b814f025 [LEADER]: Connected to new peer: Peer: permanent_uuid: "306a909ee2e344a49f9e7691e2ec8ebe" member_type: VOTER last_known_addr { host: "127.14.147.195" port: 41413 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 22:50:21.377743 17177 mvcc.cc:205] Tried to move back new op lower bound from 7334798218755493888 to 7334798218410721280. Current Snapshot: MvccSnapshot[applied={T|T < 7334798218755493888}]
I20260929 22:50:21.391568 16647 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40032:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 22:50:21.391721 16647 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40032:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 22:50:21.392552 16647 catalog_manager.cc:5782] T 00000000000000000000000000000000 P e2c35b91e0064597ac06279423fc997d: Sending DeleteTablet for 3 replicas of tablet bd40e94e78f04626b23293b5dad04ea8
I20260929 22:50:21.392917 16926 tablet_service.cc:1514] Processing DeleteTablet for tablet bd40e94e78f04626b23293b5dad04ea8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 22:50:21 UTC) from {username='kudu', principal='kudu/127.14.147.254@KRBTEST.COM'} at 127.0.0.1:48386
I20260929 22:50:21.392915 17054 tablet_service.cc:1514] Processing DeleteTablet for tablet bd40e94e78f04626b23293b5dad04ea8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 22:50:21 UTC) from {username='kudu', principal='kudu/127.14.147.254@KRBTEST.COM'} at 127.0.0.1:54066
I20260929 22:50:21.393139 16647 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40032:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 22:50:21.393290 16647 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 22:50:21.393378 16786 tablet_service.cc:1514] Processing DeleteTablet for tablet bd40e94e78f04626b23293b5dad04ea8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 22:50:21 UTC) from {username='kudu', principal='kudu/127.14.147.254@KRBTEST.COM'} at 127.0.0.1:55110
I20260929 22:50:21.393426 17194 tablet_replica.cc:330] T bd40e94e78f04626b23293b5dad04ea8 P 306a909ee2e344a49f9e7691e2ec8ebe: stopping tablet replica
I20260929 22:50:21.393623 17195 tablet_replica.cc:330] T bd40e94e78f04626b23293b5dad04ea8 P b3941f6d62734fa59b0f1005b814f025: stopping tablet replica
I20260929 22:50:21.393622 17194 raft_consensus.cc:2229] T bd40e94e78f04626b23293b5dad04ea8 P 306a909ee2e344a49f9e7691e2ec8ebe [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 22:50:21.393750 17194 raft_consensus.cc:2258] T bd40e94e78f04626b23293b5dad04ea8 P 306a909ee2e344a49f9e7691e2ec8ebe [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 22:50:21.393770 17195 raft_consensus.cc:2229] T bd40e94e78f04626b23293b5dad04ea8 P b3941f6d62734fa59b0f1005b814f025 [term 1 LEADER]: Raft consensus shutting down.
I20260929 22:50:21.393882 17195 raft_consensus.cc:2258] T bd40e94e78f04626b23293b5dad04ea8 P b3941f6d62734fa59b0f1005b814f025 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 22:50:21.393960 17193 tablet_replica.cc:330] T bd40e94e78f04626b23293b5dad04ea8 P cb60c60cc23349ab9e59004d1d2983fa: stopping tablet replica
I20260929 22:50:21.394793 17054 tablet_service.cc:1467] Processing CreateTablet for tablet 3a4e4e9e1d45404984e9e79881f7f3da (DEFAULT_TABLE table=test-table [id=6f82af8821da4857aaddea94f940e10f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 22:50:21.394793 16926 tablet_service.cc:1467] Processing CreateTablet for tablet 3a4e4e9e1d45404984e9e79881f7f3da (DEFAULT_TABLE table=test-table [id=6f82af8821da4857aaddea94f940e10f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 22:50:21.394919 16786 tablet_service.cc:1467] Processing CreateTablet for tablet 3a4e4e9e1d45404984e9e79881f7f3da (DEFAULT_TABLE table=test-table [id=6f82af8821da4857aaddea94f940e10f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 22:50:21.394971 17054 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3a4e4e9e1d45404984e9e79881f7f3da. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 22:50:21.394971 16926 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3a4e4e9e1d45404984e9e79881f7f3da. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 22:50:21.395066 16786 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3a4e4e9e1d45404984e9e79881f7f3da. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 22:50:21.396689 17153 tablet_bootstrap.cc:492] T 3a4e4e9e1d45404984e9e79881f7f3da P b3941f6d62734fa59b0f1005b814f025: Bootstrap starting.
I20260929 22:50:21.396720 17156 tablet_bootstrap.cc:492] T 3a4e4e9e1d45404984e9e79881f7f3da P 306a909ee2e344a49f9e7691e2ec8ebe: Bootstrap starting.
I20260929 22:50:21.397214 17156 tablet_bootstrap.cc:654] T 3a4e4e9e1d45404984e9e79881f7f3da P 306a909ee2e344a49f9e7691e2ec8ebe: Neither blocks nor log segments found. Creating new log.
I20260929 22:50:21.397214 17153 tablet_bootstrap.cc:654] T 3a4e4e9e1d45404984e9e79881f7f3da P b3941f6d62734fa59b0f1005b814f025: Neither blocks nor log segments found. Creating new log.
I20260929 22:50:21.397641 17152 tablet_bootstrap.cc:492] T 3a4e4e9e1d45404984e9e79881f7f3da P cb60c60cc23349ab9e59004d1d2983fa: Bootstrap starting.
I20260929 22:50:21.398082 17152 tablet_bootstrap.cc:654] T 3a4e4e9e1d45404984e9e79881f7f3da P cb60c60cc23349ab9e59004d1d2983fa: Neither blocks nor log segments found. Creating new log.
I20260929 22:50:21.398092 17156 tablet_bootstrap.cc:492] T 3a4e4e9e1d45404984e9e79881f7f3da P 306a909ee2e344a49f9e7691e2ec8ebe: No bootstrap required, opened a new log
I20260929 22:50:21.398139 17156 ts_tablet_manager.cc:1397] T 3a4e4e9e1d45404984e9e79881f7f3da P 306a909ee2e344a49f9e7691e2ec8ebe: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260929 22:50:21.398335 17156 raft_consensus.cc:348] T 3a4e4e9e1d45404984e9e79881f7f3da P 306a909ee2e344a49f9e7691e2ec8ebe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3941f6d62734fa59b0f1005b814f025" member_type: VOTER last_known_addr { host: "127.14.147.193" port: 33765 } } peers { permanent_uuid: "cb60c60cc23349ab9e59004d1d2983fa" member_type: VOTER last_known_addr { host: "127.14.147.194" port: 39385 } } peers { permanent_uuid: "306a909ee2e344a49f9e7691e2ec8ebe" member_type: VOTER last_known_addr { host: "127.14.147.195" port: 41413 } }
I20260929 22:50:21.398440 17156 raft_consensus.cc:374] T 3a4e4e9e1d45404984e9e79881f7f3da P 306a909ee2e344a49f9e7691e2ec8ebe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 22:50:21.398460 17156 raft_consensus.cc:729] T 3a4e4e9e1d45404984e9e79881f7f3da P 306a909ee2e344a49f9e7691e2ec8ebe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 306a909ee2e344a49f9e7691e2ec8ebe, State: Initialized, Role: FOLLOWER
I20260929 22:50:21.398509 17156 consensus_queue.cc:260] T 3a4e4e9e1d45404984e9e79881f7f3da P 306a909ee2e344a49f9e7691e2ec8ebe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3941f6d62734fa59b0f1005b814f025" member_type: VOTER last_known_addr { host: "127.14.147.193" port: 33765 } } peers { permanent_uuid: "cb60c60cc23349ab9e59004d1d2983fa" member_type: VOTER last_known_addr { host: "127.14.147.194" port: 39385 } } peers { permanent_uuid: "306a909ee2e344a49f9e7691e2ec8ebe" member_type: VOTER last_known_addr { host: "127.14.147.195" port: 41413 } }
I20260929 22:50:21.398602 17156 ts_tablet_manager.cc:1428] T 3a4e4e9e1d45404984e9e79881f7f3da P 306a909ee2e344a49f9e7691e2ec8ebe: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260929 22:50:21.398936 17194 ts_tablet_manager.cc:1905] T bd40e94e78f04626b23293b5dad04ea8 P 306a909ee2e344a49f9e7691e2ec8ebe: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 22:50:21.399459 17195 ts_tablet_manager.cc:1905] T bd40e94e78f04626b23293b5dad04ea8 P b3941f6d62734fa59b0f1005b814f025: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 22:50:21.401135 17194 ts_tablet_manager.cc:1918] T bd40e94e78f04626b23293b5dad04ea8 P 306a909ee2e344a49f9e7691e2ec8ebe: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260929 22:50:21.401257 17194 log.cc:1196] T bd40e94e78f04626b23293b5dad04ea8 P 306a909ee2e344a49f9e7691e2ec8ebe: Deleting WAL directory at /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790722213368738-14927-0/minicluster-data/ts-2/wal/wals/bd40e94e78f04626b23293b5dad04ea8
I20260929 22:50:21.401500 17194 ts_tablet_manager.cc:1939] T bd40e94e78f04626b23293b5dad04ea8 P 306a909ee2e344a49f9e7691e2ec8ebe: Deleting consensus metadata
I20260929 22:50:21.401714 17193 raft_consensus.cc:2229] T bd40e94e78f04626b23293b5dad04ea8 P cb60c60cc23349ab9e59004d1d2983fa [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 22:50:21.401818 17193 raft_consensus.cc:2258] T bd40e94e78f04626b23293b5dad04ea8 P cb60c60cc23349ab9e59004d1d2983fa [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 22:50:21.402093 17195 ts_tablet_manager.cc:1918] T bd40e94e78f04626b23293b5dad04ea8 P b3941f6d62734fa59b0f1005b814f025: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260929 22:50:21.402179 17195 log.cc:1196] T bd40e94e78f04626b23293b5dad04ea8 P b3941f6d62734fa59b0f1005b814f025: Deleting WAL directory at /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790722213368738-14927-0/minicluster-data/ts-0/wal/wals/bd40e94e78f04626b23293b5dad04ea8
I20260929 22:50:21.402285 16633 catalog_manager.cc:4841] TS 306a909ee2e344a49f9e7691e2ec8ebe (127.14.147.195:41413): tablet bd40e94e78f04626b23293b5dad04ea8 (table test-table [id=bedc456dc86841eba6983647aae84655]) successfully deleted
I20260929 22:50:21.402335 17193 ts_tablet_manager.cc:1905] T bd40e94e78f04626b23293b5dad04ea8 P cb60c60cc23349ab9e59004d1d2983fa: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 22:50:21.402362 17153 tablet_bootstrap.cc:492] T 3a4e4e9e1d45404984e9e79881f7f3da P b3941f6d62734fa59b0f1005b814f025: No bootstrap required, opened a new log
I20260929 22:50:21.402400 17153 ts_tablet_manager.cc:1397] T 3a4e4e9e1d45404984e9e79881f7f3da P b3941f6d62734fa59b0f1005b814f025: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.000s
I20260929 22:50:21.402608 17153 raft_consensus.cc:348] T 3a4e4e9e1d45404984e9e79881f7f3da P b3941f6d62734fa59b0f1005b814f025 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3941f6d62734fa59b0f1005b814f025" member_type: VOTER last_known_addr { host: "127.14.147.193" port: 33765 } } peers { permanent_uuid: "cb60c60cc23349ab9e59004d1d2983fa" member_type: VOTER last_known_addr { host: "127.14.147.194" port: 39385 } } peers { permanent_uuid: "306a909ee2e344a49f9e7691e2ec8ebe" member_type: VOTER last_known_addr { host: "127.14.147.195" port: 41413 } }
I20260929 22:50:21.402695 17153 raft_consensus.cc:374] T 3a4e4e9e1d45404984e9e79881f7f3da P b3941f6d62734fa59b0f1005b814f025 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 22:50:21.402712 17153 raft_consensus.cc:729] T 3a4e4e9e1d45404984e9e79881f7f3da P b3941f6d62734fa59b0f1005b814f025 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b3941f6d62734fa59b0f1005b814f025, State: Initialized, Role: FOLLOWER
I20260929 22:50:21.402755 17153 consensus_queue.cc:260] T 3a4e4e9e1d45404984e9e79881f7f3da P b3941f6d62734fa59b0f1005b814f025 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3941f6d62734fa59b0f1005b814f025" member_type: VOTER last_known_addr { host: "127.14.147.193" port: 33765 } } peers { permanent_uuid: "cb60c60cc23349ab9e59004d1d2983fa" member_type: VOTER last_known_addr { host: "127.14.147.194" port: 39385 } } peers { permanent_uuid: "306a909ee2e344a49f9e7691e2ec8ebe" member_type: VOTER last_known_addr { host: "127.14.147.195" port: 41413 } }
I20260929 22:50:21.402835 17153 ts_tablet_manager.cc:1428] T 3a4e4e9e1d45404984e9e79881f7f3da P b3941f6d62734fa59b0f1005b814f025: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260929 22:50:21.403764 17195 ts_tablet_manager.cc:1939] T bd40e94e78f04626b23293b5dad04ea8 P b3941f6d62734fa59b0f1005b814f025: Deleting consensus metadata
I20260929 22:50:21.404115 17193 ts_tablet_manager.cc:1918] T bd40e94e78f04626b23293b5dad04ea8 P cb60c60cc23349ab9e59004d1d2983fa: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260929 22:50:21.404183 17193 log.cc:1196] T bd40e94e78f04626b23293b5dad04ea8 P cb60c60cc23349ab9e59004d1d2983fa: Deleting WAL directory at /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790722213368738-14927-0/minicluster-data/ts-1/wal/wals/bd40e94e78f04626b23293b5dad04ea8
I20260929 22:50:21.404223 16633 catalog_manager.cc:4841] TS b3941f6d62734fa59b0f1005b814f025 (127.14.147.193:33765): tablet bd40e94e78f04626b23293b5dad04ea8 (table test-table [id=bedc456dc86841eba6983647aae84655]) successfully deleted
I20260929 22:50:21.404292 17152 tablet_bootstrap.cc:492] T 3a4e4e9e1d45404984e9e79881f7f3da P cb60c60cc23349ab9e59004d1d2983fa: No bootstrap required, opened a new log
I20260929 22:50:21.404336 17152 ts_tablet_manager.cc:1397] T 3a4e4e9e1d45404984e9e79881f7f3da P cb60c60cc23349ab9e59004d1d2983fa: Time spent bootstrapping tablet: real 0.007s user 0.001s sys 0.000s
I20260929 22:50:21.404408 17193 ts_tablet_manager.cc:1939] T bd40e94e78f04626b23293b5dad04ea8 P cb60c60cc23349ab9e59004d1d2983fa: Deleting consensus metadata
I20260929 22:50:21.404501 17152 raft_consensus.cc:348] T 3a4e4e9e1d45404984e9e79881f7f3da P cb60c60cc23349ab9e59004d1d2983fa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3941f6d62734fa59b0f1005b814f025" member_type: VOTER last_known_addr { host: "127.14.147.193" port: 33765 } } peers { permanent_uuid: "cb60c60cc23349ab9e59004d1d2983fa" member_type: VOTER last_known_addr { host: "127.14.147.194" port: 39385 } } peers { permanent_uuid: "306a909ee2e344a49f9e7691e2ec8ebe" member_type: VOTER last_known_addr { host: "127.14.147.195" port: 41413 } }
I20260929 22:50:21.404593 17152 raft_consensus.cc:374] T 3a4e4e9e1d45404984e9e79881f7f3da P cb60c60cc23349ab9e59004d1d2983fa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 22:50:21.404611 17152 raft_consensus.cc:729] T 3a4e4e9e1d45404984e9e79881f7f3da P cb60c60cc23349ab9e59004d1d2983fa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cb60c60cc23349ab9e59004d1d2983fa, State: Initialized, Role: FOLLOWER
I20260929 22:50:21.404656 17152 consensus_queue.cc:260] T 3a4e4e9e1d45404984e9e79881f7f3da P cb60c60cc23349ab9e59004d1d2983fa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3941f6d62734fa59b0f1005b814f025" member_type: VOTER last_known_addr { host: "127.14.147.193" port: 33765 } } peers { permanent_uuid: "cb60c60cc23349ab9e59004d1d2983fa" member_type: VOTER last_known_addr { host: "127.14.147.194" port: 39385 } } peers { permanent_uuid: "306a909ee2e344a49f9e7691e2ec8ebe" member_type: VOTER last_known_addr { host: "127.14.147.195" port: 41413 } }
I20260929 22:50:21.404743 17152 ts_tablet_manager.cc:1428] T 3a4e4e9e1d45404984e9e79881f7f3da P cb60c60cc23349ab9e59004d1d2983fa: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260929 22:50:21.404793 16635 catalog_manager.cc:4841] TS cb60c60cc23349ab9e59004d1d2983fa (127.14.147.194:39385): tablet bd40e94e78f04626b23293b5dad04ea8 (table test-table [id=bedc456dc86841eba6983647aae84655]) successfully deleted
W20260929 22:50:21.416029 17133 tablet.cc:2369] T 3a4e4e9e1d45404984e9e79881f7f3da P 306a909ee2e344a49f9e7691e2ec8ebe: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 22:50:21.649024 17162 raft_consensus.cc:482] T 3a4e4e9e1d45404984e9e79881f7f3da P b3941f6d62734fa59b0f1005b814f025 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 22:50:21.649137 17162 raft_consensus.cc:504] T 3a4e4e9e1d45404984e9e79881f7f3da P b3941f6d62734fa59b0f1005b814f025 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3941f6d62734fa59b0f1005b814f025" member_type: VOTER last_known_addr { host: "127.14.147.193" port: 33765 } } peers { permanent_uuid: "cb60c60cc23349ab9e59004d1d2983fa" member_type: VOTER last_known_addr { host: "127.14.147.194" port: 39385 } } peers { permanent_uuid: "306a909ee2e344a49f9e7691e2ec8ebe" member_type: VOTER last_known_addr { host: "127.14.147.195" port: 41413 } }
I20260929 22:50:21.649313 17162 leader_election.cc:288] T 3a4e4e9e1d45404984e9e79881f7f3da P b3941f6d62734fa59b0f1005b814f025 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cb60c60cc23349ab9e59004d1d2983fa (127.14.147.194:39385), 306a909ee2e344a49f9e7691e2ec8ebe (127.14.147.195:41413)
I20260929 22:50:21.649555 17086 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3a4e4e9e1d45404984e9e79881f7f3da" candidate_uuid: "b3941f6d62734fa59b0f1005b814f025" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "306a909ee2e344a49f9e7691e2ec8ebe" is_pre_election: true
I20260929 22:50:21.649668 17086 raft_consensus.cc:2454] T 3a4e4e9e1d45404984e9e79881f7f3da P 306a909ee2e344a49f9e7691e2ec8ebe [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b3941f6d62734fa59b0f1005b814f025 in term 0.
I20260929 22:50:21.649678 16946 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3a4e4e9e1d45404984e9e79881f7f3da" candidate_uuid: "b3941f6d62734fa59b0f1005b814f025" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cb60c60cc23349ab9e59004d1d2983fa" is_pre_election: true
I20260929 22:50:21.649821 16738 leader_election.cc:302] T 3a4e4e9e1d45404984e9e79881f7f3da P b3941f6d62734fa59b0f1005b814f025 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 306a909ee2e344a49f9e7691e2ec8ebe, b3941f6d62734fa59b0f1005b814f025; no voters:
I20260929 22:50:21.649861 16946 raft_consensus.cc:2454] T 3a4e4e9e1d45404984e9e79881f7f3da P cb60c60cc23349ab9e59004d1d2983fa [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b3941f6d62734fa59b0f1005b814f025 in term 0.
I20260929 22:50:21.649909 17162 raft_consensus.cc:2781] T 3a4e4e9e1d45404984e9e79881f7f3da P b3941f6d62734fa59b0f1005b814f025 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 22:50:21.649957 17162 raft_consensus.cc:482] T 3a4e4e9e1d45404984e9e79881f7f3da P b3941f6d62734fa59b0f1005b814f025 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 22:50:21.650003 17162 raft_consensus.cc:3037] T 3a4e4e9e1d45404984e9e79881f7f3da P b3941f6d62734fa59b0f1005b814f025 [term 0 FOLLOWER]: Advancing to term 1
I20260929 22:50:21.650767 17162 raft_consensus.cc:504] T 3a4e4e9e1d45404984e9e79881f7f3da P b3941f6d62734fa59b0f1005b814f025 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3941f6d62734fa59b0f1005b814f025" member_type: VOTER last_known_addr { host: "127.14.147.193" port: 33765 } } peers { permanent_uuid: "cb60c60cc23349ab9e59004d1d2983fa" member_type: VOTER last_known_addr { host: "127.14.147.194" port: 39385 } } peers { permanent_uuid: "306a909ee2e344a49f9e7691e2ec8ebe" member_type: VOTER last_known_addr { host: "127.14.147.195" port: 41413 } }
I20260929 22:50:21.650950 17162 leader_election.cc:288] T 3a4e4e9e1d45404984e9e79881f7f3da P b3941f6d62734fa59b0f1005b814f025 [CANDIDATE]: Term 1 election: Requested vote from peers cb60c60cc23349ab9e59004d1d2983fa (127.14.147.194:39385), 306a909ee2e344a49f9e7691e2ec8ebe (127.14.147.195:41413)
I20260929 22:50:21.651147 16946 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3a4e4e9e1d45404984e9e79881f7f3da" candidate_uuid: "b3941f6d62734fa59b0f1005b814f025" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cb60c60cc23349ab9e59004d1d2983fa"
I20260929 22:50:21.651175 17086 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3a4e4e9e1d45404984e9e79881f7f3da" candidate_uuid: "b3941f6d62734fa59b0f1005b814f025" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "306a909ee2e344a49f9e7691e2ec8ebe"
I20260929 22:50:21.651247 17086 raft_consensus.cc:3037] T 3a4e4e9e1d45404984e9e79881f7f3da P 306a909ee2e344a49f9e7691e2ec8ebe [term 0 FOLLOWER]: Advancing to term 1
I20260929 22:50:21.651250 16946 raft_consensus.cc:3037] T 3a4e4e9e1d45404984e9e79881f7f3da P cb60c60cc23349ab9e59004d1d2983fa [term 0 FOLLOWER]: Advancing to term 1
I20260929 22:50:21.652086 17086 raft_consensus.cc:2454] T 3a4e4e9e1d45404984e9e79881f7f3da P 306a909ee2e344a49f9e7691e2ec8ebe [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b3941f6d62734fa59b0f1005b814f025 in term 1.
I20260929 22:50:21.652086 16946 raft_consensus.cc:2454] T 3a4e4e9e1d45404984e9e79881f7f3da P cb60c60cc23349ab9e59004d1d2983fa [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b3941f6d62734fa59b0f1005b814f025 in term 1.
I20260929 22:50:21.652308 16738 leader_election.cc:302] T 3a4e4e9e1d45404984e9e79881f7f3da P b3941f6d62734fa59b0f1005b814f025 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 306a909ee2e344a49f9e7691e2ec8ebe, b3941f6d62734fa59b0f1005b814f025; no voters:
I20260929 22:50:21.652400 17162 raft_consensus.cc:2781] T 3a4e4e9e1d45404984e9e79881f7f3da P b3941f6d62734fa59b0f1005b814f025 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 22:50:21.652450 17162 raft_consensus.cc:686] T 3a4e4e9e1d45404984e9e79881f7f3da P b3941f6d62734fa59b0f1005b814f025 [term 1 LEADER]: Becoming Leader. State: Replica: b3941f6d62734fa59b0f1005b814f025, State: Running, Role: LEADER
I20260929 22:50:21.652498 17162 consensus_queue.cc:237] T 3a4e4e9e1d45404984e9e79881f7f3da P b3941f6d62734fa59b0f1005b814f025 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3941f6d62734fa59b0f1005b814f025" member_type: VOTER last_known_addr { host: "127.14.147.193" port: 33765 } } peers { permanent_uuid: "cb60c60cc23349ab9e59004d1d2983fa" member_type: VOTER last_known_addr { host: "127.14.147.194" port: 39385 } } peers { permanent_uuid: "306a909ee2e344a49f9e7691e2ec8ebe" member_type: VOTER last_known_addr { host: "127.14.147.195" port: 41413 } }
I20260929 22:50:21.653096 16647 catalog_manager.cc:5495] T 3a4e4e9e1d45404984e9e79881f7f3da P b3941f6d62734fa59b0f1005b814f025 reported cstate change: term changed from 0 to 1, leader changed from <none> to b3941f6d62734fa59b0f1005b814f025 (127.14.147.193). New cstate: current_term: 1 leader_uuid: "b3941f6d62734fa59b0f1005b814f025" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3941f6d62734fa59b0f1005b814f025" member_type: VOTER last_known_addr { host: "127.14.147.193" port: 33765 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cb60c60cc23349ab9e59004d1d2983fa" member_type: VOTER last_known_addr { host: "127.14.147.194" port: 39385 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "306a909ee2e344a49f9e7691e2ec8ebe" member_type: VOTER last_known_addr { host: "127.14.147.195" port: 41413 } health_report { overall_health: UNKNOWN } } }
I20260929 22:50:21.693288 16946 raft_consensus.cc:1261] T 3a4e4e9e1d45404984e9e79881f7f3da P cb60c60cc23349ab9e59004d1d2983fa [term 1 FOLLOWER]: Refusing update from remote peer b3941f6d62734fa59b0f1005b814f025: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 22:50:21.693336 17086 raft_consensus.cc:1261] T 3a4e4e9e1d45404984e9e79881f7f3da P 306a909ee2e344a49f9e7691e2ec8ebe [term 1 FOLLOWER]: Refusing update from remote peer b3941f6d62734fa59b0f1005b814f025: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 22:50:21.693547 17191 consensus_queue.cc:1035] T 3a4e4e9e1d45404984e9e79881f7f3da P b3941f6d62734fa59b0f1005b814f025 [LEADER]: Connected to new peer: Peer: permanent_uuid: "306a909ee2e344a49f9e7691e2ec8ebe" member_type: VOTER last_known_addr { host: "127.14.147.195" port: 41413 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 22:50:21.693750 17191 consensus_queue.cc:1035] T 3a4e4e9e1d45404984e9e79881f7f3da P b3941f6d62734fa59b0f1005b814f025 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cb60c60cc23349ab9e59004d1d2983fa" member_type: VOTER last_known_addr { host: "127.14.147.194" port: 39385 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260929 22:50:21.698825 17177 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260929 22:50:23.217135 16647 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40032:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 22:50:23.217237 16647 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40032:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 22:50:23.218111 16647 catalog_manager.cc:5782] T 00000000000000000000000000000000 P e2c35b91e0064597ac06279423fc997d: Sending DeleteTablet for 3 replicas of tablet 3a4e4e9e1d45404984e9e79881f7f3da
I20260929 22:50:23.218365 16786 tablet_service.cc:1514] Processing DeleteTablet for tablet 3a4e4e9e1d45404984e9e79881f7f3da with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 22:50:23 UTC) from {username='kudu', principal='kudu/127.14.147.254@KRBTEST.COM'} at 127.0.0.1:55110
I20260929 22:50:23.218420 17054 tablet_service.cc:1514] Processing DeleteTablet for tablet 3a4e4e9e1d45404984e9e79881f7f3da with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 22:50:23 UTC) from {username='kudu', principal='kudu/127.14.147.254@KRBTEST.COM'} at 127.0.0.1:54066
I20260929 22:50:23.218569 17221 tablet_replica.cc:330] T 3a4e4e9e1d45404984e9e79881f7f3da P b3941f6d62734fa59b0f1005b814f025: stopping tablet replica
I20260929 22:50:23.218555 16926 tablet_service.cc:1514] Processing DeleteTablet for tablet 3a4e4e9e1d45404984e9e79881f7f3da with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 22:50:23 UTC) from {username='kudu', principal='kudu/127.14.147.254@KRBTEST.COM'} at 127.0.0.1:48386
I20260929 22:50:23.218622 17221 raft_consensus.cc:2229] T 3a4e4e9e1d45404984e9e79881f7f3da P b3941f6d62734fa59b0f1005b814f025 [term 1 LEADER]: Raft consensus shutting down.
I20260929 22:50:23.218672 17221 raft_consensus.cc:2258] T 3a4e4e9e1d45404984e9e79881f7f3da P b3941f6d62734fa59b0f1005b814f025 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 22:50:23.218748 17223 tablet_replica.cc:330] T 3a4e4e9e1d45404984e9e79881f7f3da P cb60c60cc23349ab9e59004d1d2983fa: stopping tablet replica
I20260929 22:50:23.218796 17223 raft_consensus.cc:2229] T 3a4e4e9e1d45404984e9e79881f7f3da P cb60c60cc23349ab9e59004d1d2983fa [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 22:50:23.218827 17223 raft_consensus.cc:2258] T 3a4e4e9e1d45404984e9e79881f7f3da P cb60c60cc23349ab9e59004d1d2983fa [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 22:50:23.218909 17221 ts_tablet_manager.cc:1905] T 3a4e4e9e1d45404984e9e79881f7f3da P b3941f6d62734fa59b0f1005b814f025: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 22:50:23.220449 17222 tablet_replica.cc:330] T 3a4e4e9e1d45404984e9e79881f7f3da P 306a909ee2e344a49f9e7691e2ec8ebe: stopping tablet replica
I20260929 22:50:23.220577 17223 ts_tablet_manager.cc:1905] T 3a4e4e9e1d45404984e9e79881f7f3da P cb60c60cc23349ab9e59004d1d2983fa: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 22:50:23.220625 17221 ts_tablet_manager.cc:1918] T 3a4e4e9e1d45404984e9e79881f7f3da P b3941f6d62734fa59b0f1005b814f025: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260929 22:50:23.220679 17221 log.cc:1196] T 3a4e4e9e1d45404984e9e79881f7f3da P b3941f6d62734fa59b0f1005b814f025: Deleting WAL directory at /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790722213368738-14927-0/minicluster-data/ts-0/wal/wals/3a4e4e9e1d45404984e9e79881f7f3da
I20260929 22:50:23.220929 17221 ts_tablet_manager.cc:1939] T 3a4e4e9e1d45404984e9e79881f7f3da P b3941f6d62734fa59b0f1005b814f025: Deleting consensus metadata
I20260929 22:50:23.221215 17222 raft_consensus.cc:2229] T 3a4e4e9e1d45404984e9e79881f7f3da P 306a909ee2e344a49f9e7691e2ec8ebe [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 22:50:23.221228 16633 catalog_manager.cc:4841] TS b3941f6d62734fa59b0f1005b814f025 (127.14.147.193:33765): tablet 3a4e4e9e1d45404984e9e79881f7f3da (table test-table [id=6f82af8821da4857aaddea94f940e10f]) successfully deleted
I20260929 22:50:23.221277 17222 raft_consensus.cc:2258] T 3a4e4e9e1d45404984e9e79881f7f3da P 306a909ee2e344a49f9e7691e2ec8ebe [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 22:50:23.221529 17222 ts_tablet_manager.cc:1905] T 3a4e4e9e1d45404984e9e79881f7f3da P 306a909ee2e344a49f9e7691e2ec8ebe: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 22:50:23.221969 17223 ts_tablet_manager.cc:1918] T 3a4e4e9e1d45404984e9e79881f7f3da P cb60c60cc23349ab9e59004d1d2983fa: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260929 22:50:23.222030 17223 log.cc:1196] T 3a4e4e9e1d45404984e9e79881f7f3da P cb60c60cc23349ab9e59004d1d2983fa: Deleting WAL directory at /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790722213368738-14927-0/minicluster-data/ts-1/wal/wals/3a4e4e9e1d45404984e9e79881f7f3da
I20260929 22:50:23.222204 17223 ts_tablet_manager.cc:1939] T 3a4e4e9e1d45404984e9e79881f7f3da P cb60c60cc23349ab9e59004d1d2983fa: Deleting consensus metadata
I20260929 22:50:23.222625 16635 catalog_manager.cc:4841] TS cb60c60cc23349ab9e59004d1d2983fa (127.14.147.194:39385): tablet 3a4e4e9e1d45404984e9e79881f7f3da (table test-table [id=6f82af8821da4857aaddea94f940e10f]) successfully deleted
I20260929 22:50:23.223110 17222 ts_tablet_manager.cc:1918] T 3a4e4e9e1d45404984e9e79881f7f3da P 306a909ee2e344a49f9e7691e2ec8ebe: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260929 22:50:23.223170 17222 log.cc:1196] T 3a4e4e9e1d45404984e9e79881f7f3da P 306a909ee2e344a49f9e7691e2ec8ebe: Deleting WAL directory at /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790722213368738-14927-0/minicluster-data/ts-2/wal/wals/3a4e4e9e1d45404984e9e79881f7f3da
I20260929 22:50:23.223366 17222 ts_tablet_manager.cc:1939] T 3a4e4e9e1d45404984e9e79881f7f3da P 306a909ee2e344a49f9e7691e2ec8ebe: Deleting consensus metadata
I20260929 22:50:23.223702 16633 catalog_manager.cc:4841] TS 306a909ee2e344a49f9e7691e2ec8ebe (127.14.147.195:41413): tablet 3a4e4e9e1d45404984e9e79881f7f3da (table test-table [id=6f82af8821da4857aaddea94f940e10f]) successfully deleted
Sep 29 22:50:23 dist-test-slave-98mk krb5kdc[16603](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790722223, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 29 22:50:23 dist-test-slave-98mk krb5kdc[16603](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790722223, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.14.147.254@KRBTEST.COM
I20260929 22:50:23.241647 17231 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:50:23.235128 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40054 (local address 127.14.147.254:35089)
0929 22:50:23.235272 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0929 22:50:23.235274 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 22:50:23.235282 (+ 8us) server_negotiation.cc:407] Connection header received
0929 22:50:23.235307 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 22:50:23.235308 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 22:50:23.235318 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 22:50:23.235384 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0929 22:50:23.235989 (+ 605us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:50:23.236512 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:50:23.237236 (+ 724us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:50:23.237422 (+ 186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:50:23.237465 (+ 43us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 22:50:23.239524 (+ 2059us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 22:50:23.239525 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 22:50:23.239525 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 22:50:23.239532 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0929 22:50:23.240784 (+ 1252us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 22:50:23.241089 (+ 305us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 22:50:23.241090 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 22:50:23.241090 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 22:50:23.241144 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 22:50:23.241271 (+ 127us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 22:50:23.241271 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 22:50:23.241271 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 22:50:23.241371 (+ 100us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 22:50:23.241439 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0929 22:50:23.241510 (+ 71us) server_negotiation.cc:299] Negotiation successful
0929 22:50:23.241571 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":120,"thread_start_us":84,"threads_started":1}
I20260929 22:50:23.242949 16647 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:40054:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 22:50:23.243096 16647 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 22:50:23.244333 16786 tablet_service.cc:1467] Processing CreateTablet for tablet d616ea7c990c4556a2782e0fde1519a2 (DEFAULT_TABLE table=test-table [id=abc1be8a5ce54caf94ca18f938dce24c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 22:50:23.244342 17054 tablet_service.cc:1467] Processing CreateTablet for tablet d616ea7c990c4556a2782e0fde1519a2 (DEFAULT_TABLE table=test-table [id=abc1be8a5ce54caf94ca18f938dce24c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 22:50:23.244429 16926 tablet_service.cc:1467] Processing CreateTablet for tablet d616ea7c990c4556a2782e0fde1519a2 (DEFAULT_TABLE table=test-table [id=abc1be8a5ce54caf94ca18f938dce24c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 22:50:23.244477 17054 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d616ea7c990c4556a2782e0fde1519a2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 22:50:23.244478 16786 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d616ea7c990c4556a2782e0fde1519a2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 22:50:23.244540 16926 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d616ea7c990c4556a2782e0fde1519a2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 22:50:23.245807 17233 tablet_bootstrap.cc:492] T d616ea7c990c4556a2782e0fde1519a2 P cb60c60cc23349ab9e59004d1d2983fa: Bootstrap starting.
I20260929 22:50:23.245821 17234 tablet_bootstrap.cc:492] T d616ea7c990c4556a2782e0fde1519a2 P 306a909ee2e344a49f9e7691e2ec8ebe: Bootstrap starting.
I20260929 22:50:23.246096 17232 tablet_bootstrap.cc:492] T d616ea7c990c4556a2782e0fde1519a2 P b3941f6d62734fa59b0f1005b814f025: Bootstrap starting.
I20260929 22:50:23.246232 17234 tablet_bootstrap.cc:654] T d616ea7c990c4556a2782e0fde1519a2 P 306a909ee2e344a49f9e7691e2ec8ebe: Neither blocks nor log segments found. Creating new log.
I20260929 22:50:23.246232 17233 tablet_bootstrap.cc:654] T d616ea7c990c4556a2782e0fde1519a2 P cb60c60cc23349ab9e59004d1d2983fa: Neither blocks nor log segments found. Creating new log.
I20260929 22:50:23.246459 17232 tablet_bootstrap.cc:654] T d616ea7c990c4556a2782e0fde1519a2 P b3941f6d62734fa59b0f1005b814f025: Neither blocks nor log segments found. Creating new log.
I20260929 22:50:23.246918 17234 tablet_bootstrap.cc:492] T d616ea7c990c4556a2782e0fde1519a2 P 306a909ee2e344a49f9e7691e2ec8ebe: No bootstrap required, opened a new log
I20260929 22:50:23.246971 17234 ts_tablet_manager.cc:1397] T d616ea7c990c4556a2782e0fde1519a2 P 306a909ee2e344a49f9e7691e2ec8ebe: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260929 22:50:23.247015 17232 tablet_bootstrap.cc:492] T d616ea7c990c4556a2782e0fde1519a2 P b3941f6d62734fa59b0f1005b814f025: No bootstrap required, opened a new log
I20260929 22:50:23.247038 17233 tablet_bootstrap.cc:492] T d616ea7c990c4556a2782e0fde1519a2 P cb60c60cc23349ab9e59004d1d2983fa: No bootstrap required, opened a new log
I20260929 22:50:23.247064 17232 ts_tablet_manager.cc:1397] T d616ea7c990c4556a2782e0fde1519a2 P b3941f6d62734fa59b0f1005b814f025: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260929 22:50:23.247095 17233 ts_tablet_manager.cc:1397] T d616ea7c990c4556a2782e0fde1519a2 P cb60c60cc23349ab9e59004d1d2983fa: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260929 22:50:23.247128 17234 raft_consensus.cc:348] T d616ea7c990c4556a2782e0fde1519a2 P 306a909ee2e344a49f9e7691e2ec8ebe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3941f6d62734fa59b0f1005b814f025" member_type: VOTER last_known_addr { host: "127.14.147.193" port: 33765 } } peers { permanent_uuid: "cb60c60cc23349ab9e59004d1d2983fa" member_type: VOTER last_known_addr { host: "127.14.147.194" port: 39385 } } peers { permanent_uuid: "306a909ee2e344a49f9e7691e2ec8ebe" member_type: VOTER last_known_addr { host: "127.14.147.195" port: 41413 } }
I20260929 22:50:23.247200 17234 raft_consensus.cc:374] T d616ea7c990c4556a2782e0fde1519a2 P 306a909ee2e344a49f9e7691e2ec8ebe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 22:50:23.247174 17232 raft_consensus.cc:348] T d616ea7c990c4556a2782e0fde1519a2 P b3941f6d62734fa59b0f1005b814f025 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3941f6d62734fa59b0f1005b814f025" member_type: VOTER last_known_addr { host: "127.14.147.193" port: 33765 } } peers { permanent_uuid: "cb60c60cc23349ab9e59004d1d2983fa" member_type: VOTER last_known_addr { host: "127.14.147.194" port: 39385 } } peers { permanent_uuid: "306a909ee2e344a49f9e7691e2ec8ebe" member_type: VOTER last_known_addr { host: "127.14.147.195" port: 41413 } }
I20260929 22:50:23.247216 17234 raft_consensus.cc:729] T d616ea7c990c4556a2782e0fde1519a2 P 306a909ee2e344a49f9e7691e2ec8ebe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 306a909ee2e344a49f9e7691e2ec8ebe, State: Initialized, Role: FOLLOWER
I20260929 22:50:23.247224 17232 raft_consensus.cc:374] T d616ea7c990c4556a2782e0fde1519a2 P b3941f6d62734fa59b0f1005b814f025 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 22:50:23.247236 17232 raft_consensus.cc:729] T d616ea7c990c4556a2782e0fde1519a2 P b3941f6d62734fa59b0f1005b814f025 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b3941f6d62734fa59b0f1005b814f025, State: Initialized, Role: FOLLOWER
I20260929 22:50:23.247242 17233 raft_consensus.cc:348] T d616ea7c990c4556a2782e0fde1519a2 P cb60c60cc23349ab9e59004d1d2983fa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3941f6d62734fa59b0f1005b814f025" member_type: VOTER last_known_addr { host: "127.14.147.193" port: 33765 } } peers { permanent_uuid: "cb60c60cc23349ab9e59004d1d2983fa" member_type: VOTER last_known_addr { host: "127.14.147.194" port: 39385 } } peers { permanent_uuid: "306a909ee2e344a49f9e7691e2ec8ebe" member_type: VOTER last_known_addr { host: "127.14.147.195" port: 41413 } }
I20260929 22:50:23.247262 17234 consensus_queue.cc:260] T d616ea7c990c4556a2782e0fde1519a2 P 306a909ee2e344a49f9e7691e2ec8ebe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3941f6d62734fa59b0f1005b814f025" member_type: VOTER last_known_addr { host: "127.14.147.193" port: 33765 } } peers { permanent_uuid: "cb60c60cc23349ab9e59004d1d2983fa" member_type: VOTER last_known_addr { host: "127.14.147.194" port: 39385 } } peers { permanent_uuid: "306a909ee2e344a49f9e7691e2ec8ebe" member_type: VOTER last_known_addr { host: "127.14.147.195" port: 41413 } }
I20260929 22:50:23.247303 17233 raft_consensus.cc:374] T d616ea7c990c4556a2782e0fde1519a2 P cb60c60cc23349ab9e59004d1d2983fa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 22:50:23.247288 17232 consensus_queue.cc:260] T d616ea7c990c4556a2782e0fde1519a2 P b3941f6d62734fa59b0f1005b814f025 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3941f6d62734fa59b0f1005b814f025" member_type: VOTER last_known_addr { host: "127.14.147.193" port: 33765 } } peers { permanent_uuid: "cb60c60cc23349ab9e59004d1d2983fa" member_type: VOTER last_known_addr { host: "127.14.147.194" port: 39385 } } peers { permanent_uuid: "306a909ee2e344a49f9e7691e2ec8ebe" member_type: VOTER last_known_addr { host: "127.14.147.195" port: 41413 } }
I20260929 22:50:23.247321 17233 raft_consensus.cc:729] T d616ea7c990c4556a2782e0fde1519a2 P cb60c60cc23349ab9e59004d1d2983fa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cb60c60cc23349ab9e59004d1d2983fa, State: Initialized, Role: FOLLOWER
I20260929 22:50:23.247366 17233 consensus_queue.cc:260] T d616ea7c990c4556a2782e0fde1519a2 P cb60c60cc23349ab9e59004d1d2983fa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3941f6d62734fa59b0f1005b814f025" member_type: VOTER last_known_addr { host: "127.14.147.193" port: 33765 } } peers { permanent_uuid: "cb60c60cc23349ab9e59004d1d2983fa" member_type: VOTER last_known_addr { host: "127.14.147.194" port: 39385 } } peers { permanent_uuid: "306a909ee2e344a49f9e7691e2ec8ebe" member_type: VOTER last_known_addr { host: "127.14.147.195" port: 41413 } }
I20260929 22:50:23.247406 17232 ts_tablet_manager.cc:1428] T d616ea7c990c4556a2782e0fde1519a2 P b3941f6d62734fa59b0f1005b814f025: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260929 22:50:23.247448 17234 ts_tablet_manager.cc:1428] T d616ea7c990c4556a2782e0fde1519a2 P 306a909ee2e344a49f9e7691e2ec8ebe: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260929 22:50:23.247545 17233 ts_tablet_manager.cc:1428] T d616ea7c990c4556a2782e0fde1519a2 P cb60c60cc23349ab9e59004d1d2983fa: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260929 22:50:23.501446 17204 raft_consensus.cc:482] T d616ea7c990c4556a2782e0fde1519a2 P b3941f6d62734fa59b0f1005b814f025 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 22:50:23.501556 17204 raft_consensus.cc:504] T d616ea7c990c4556a2782e0fde1519a2 P b3941f6d62734fa59b0f1005b814f025 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3941f6d62734fa59b0f1005b814f025" member_type: VOTER last_known_addr { host: "127.14.147.193" port: 33765 } } peers { permanent_uuid: "cb60c60cc23349ab9e59004d1d2983fa" member_type: VOTER last_known_addr { host: "127.14.147.194" port: 39385 } } peers { permanent_uuid: "306a909ee2e344a49f9e7691e2ec8ebe" member_type: VOTER last_known_addr { host: "127.14.147.195" port: 41413 } }
I20260929 22:50:23.501750 17204 leader_election.cc:288] T d616ea7c990c4556a2782e0fde1519a2 P b3941f6d62734fa59b0f1005b814f025 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cb60c60cc23349ab9e59004d1d2983fa (127.14.147.194:39385), 306a909ee2e344a49f9e7691e2ec8ebe (127.14.147.195:41413)
I20260929 22:50:23.502013 16946 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d616ea7c990c4556a2782e0fde1519a2" candidate_uuid: "b3941f6d62734fa59b0f1005b814f025" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cb60c60cc23349ab9e59004d1d2983fa" is_pre_election: true
I20260929 22:50:23.502012 17086 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d616ea7c990c4556a2782e0fde1519a2" candidate_uuid: "b3941f6d62734fa59b0f1005b814f025" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "306a909ee2e344a49f9e7691e2ec8ebe" is_pre_election: true
I20260929 22:50:23.502097 16946 raft_consensus.cc:2454] T d616ea7c990c4556a2782e0fde1519a2 P cb60c60cc23349ab9e59004d1d2983fa [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b3941f6d62734fa59b0f1005b814f025 in term 0.
I20260929 22:50:23.502125 17086 raft_consensus.cc:2454] T d616ea7c990c4556a2782e0fde1519a2 P 306a909ee2e344a49f9e7691e2ec8ebe [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b3941f6d62734fa59b0f1005b814f025 in term 0.
I20260929 22:50:23.502274 16740 leader_election.cc:302] T d616ea7c990c4556a2782e0fde1519a2 P b3941f6d62734fa59b0f1005b814f025 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b3941f6d62734fa59b0f1005b814f025, cb60c60cc23349ab9e59004d1d2983fa; no voters:
I20260929 22:50:23.502372 17204 raft_consensus.cc:2781] T d616ea7c990c4556a2782e0fde1519a2 P b3941f6d62734fa59b0f1005b814f025 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 22:50:23.502419 17204 raft_consensus.cc:482] T d616ea7c990c4556a2782e0fde1519a2 P b3941f6d62734fa59b0f1005b814f025 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 22:50:23.502439 17204 raft_consensus.cc:3037] T d616ea7c990c4556a2782e0fde1519a2 P b3941f6d62734fa59b0f1005b814f025 [term 0 FOLLOWER]: Advancing to term 1
I20260929 22:50:23.503136 17204 raft_consensus.cc:504] T d616ea7c990c4556a2782e0fde1519a2 P b3941f6d62734fa59b0f1005b814f025 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3941f6d62734fa59b0f1005b814f025" member_type: VOTER last_known_addr { host: "127.14.147.193" port: 33765 } } peers { permanent_uuid: "cb60c60cc23349ab9e59004d1d2983fa" member_type: VOTER last_known_addr { host: "127.14.147.194" port: 39385 } } peers { permanent_uuid: "306a909ee2e344a49f9e7691e2ec8ebe" member_type: VOTER last_known_addr { host: "127.14.147.195" port: 41413 } }
I20260929 22:50:23.503255 17204 leader_election.cc:288] T d616ea7c990c4556a2782e0fde1519a2 P b3941f6d62734fa59b0f1005b814f025 [CANDIDATE]: Term 1 election: Requested vote from peers cb60c60cc23349ab9e59004d1d2983fa (127.14.147.194:39385), 306a909ee2e344a49f9e7691e2ec8ebe (127.14.147.195:41413)
I20260929 22:50:23.503398 17086 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d616ea7c990c4556a2782e0fde1519a2" candidate_uuid: "b3941f6d62734fa59b0f1005b814f025" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "306a909ee2e344a49f9e7691e2ec8ebe"
I20260929 22:50:23.503448 16946 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d616ea7c990c4556a2782e0fde1519a2" candidate_uuid: "b3941f6d62734fa59b0f1005b814f025" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cb60c60cc23349ab9e59004d1d2983fa"
I20260929 22:50:23.503501 17086 raft_consensus.cc:3037] T d616ea7c990c4556a2782e0fde1519a2 P 306a909ee2e344a49f9e7691e2ec8ebe [term 0 FOLLOWER]: Advancing to term 1
I20260929 22:50:23.503512 16946 raft_consensus.cc:3037] T d616ea7c990c4556a2782e0fde1519a2 P cb60c60cc23349ab9e59004d1d2983fa [term 0 FOLLOWER]: Advancing to term 1
I20260929 22:50:23.504083 17086 raft_consensus.cc:2454] T d616ea7c990c4556a2782e0fde1519a2 P 306a909ee2e344a49f9e7691e2ec8ebe [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b3941f6d62734fa59b0f1005b814f025 in term 1.
I20260929 22:50:23.504105 16946 raft_consensus.cc:2454] T d616ea7c990c4556a2782e0fde1519a2 P cb60c60cc23349ab9e59004d1d2983fa [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b3941f6d62734fa59b0f1005b814f025 in term 1.
I20260929 22:50:23.504249 16738 leader_election.cc:302] T d616ea7c990c4556a2782e0fde1519a2 P b3941f6d62734fa59b0f1005b814f025 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 306a909ee2e344a49f9e7691e2ec8ebe, b3941f6d62734fa59b0f1005b814f025; no voters:
I20260929 22:50:23.504340 17204 raft_consensus.cc:2781] T d616ea7c990c4556a2782e0fde1519a2 P b3941f6d62734fa59b0f1005b814f025 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 22:50:23.504379 17204 raft_consensus.cc:686] T d616ea7c990c4556a2782e0fde1519a2 P b3941f6d62734fa59b0f1005b814f025 [term 1 LEADER]: Becoming Leader. State: Replica: b3941f6d62734fa59b0f1005b814f025, State: Running, Role: LEADER
I20260929 22:50:23.504421 17204 consensus_queue.cc:237] T d616ea7c990c4556a2782e0fde1519a2 P b3941f6d62734fa59b0f1005b814f025 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3941f6d62734fa59b0f1005b814f025" member_type: VOTER last_known_addr { host: "127.14.147.193" port: 33765 } } peers { permanent_uuid: "cb60c60cc23349ab9e59004d1d2983fa" member_type: VOTER last_known_addr { host: "127.14.147.194" port: 39385 } } peers { permanent_uuid: "306a909ee2e344a49f9e7691e2ec8ebe" member_type: VOTER last_known_addr { host: "127.14.147.195" port: 41413 } }
I20260929 22:50:23.505031 16646 catalog_manager.cc:5495] T d616ea7c990c4556a2782e0fde1519a2 P b3941f6d62734fa59b0f1005b814f025 reported cstate change: term changed from 0 to 1, leader changed from <none> to b3941f6d62734fa59b0f1005b814f025 (127.14.147.193). New cstate: current_term: 1 leader_uuid: "b3941f6d62734fa59b0f1005b814f025" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3941f6d62734fa59b0f1005b814f025" member_type: VOTER last_known_addr { host: "127.14.147.193" port: 33765 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cb60c60cc23349ab9e59004d1d2983fa" member_type: VOTER last_known_addr { host: "127.14.147.194" port: 39385 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "306a909ee2e344a49f9e7691e2ec8ebe" member_type: VOTER last_known_addr { host: "127.14.147.195" port: 41413 } health_report { overall_health: UNKNOWN } } }
I20260929 22:50:23.535259 17242 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:50:23.532970 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55142 (local address 127.14.147.193:33765)
0929 22:50:23.533110 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0929 22:50:23.533112 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 22:50:23.533120 (+ 8us) server_negotiation.cc:407] Connection header received
0929 22:50:23.533142 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 22:50:23.533143 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 22:50:23.533153 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 22:50:23.533210 (+ 57us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 22:50:23.533317 (+ 107us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:50:23.533858 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:50:23.534658 (+ 800us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:50:23.534882 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:50:23.534949 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 22:50:23.535039 (+ 90us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 22:50:23.535067 (+ 28us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 22:50:23.535100 (+ 33us) server_negotiation.cc:1035] Waiting for connection context
0929 22:50:23.535147 (+ 47us) server_negotiation.cc:299] Negotiation successful
0929 22:50:23.535169 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":114,"thread_start_us":81,"threads_started":1}
I20260929 22:50:23.537477 17086 raft_consensus.cc:1261] T d616ea7c990c4556a2782e0fde1519a2 P 306a909ee2e344a49f9e7691e2ec8ebe [term 1 FOLLOWER]: Refusing update from remote peer b3941f6d62734fa59b0f1005b814f025: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 22:50:23.537477 16946 raft_consensus.cc:1261] T d616ea7c990c4556a2782e0fde1519a2 P cb60c60cc23349ab9e59004d1d2983fa [term 1 FOLLOWER]: Refusing update from remote peer b3941f6d62734fa59b0f1005b814f025: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 22:50:23.537706 17204 consensus_queue.cc:1035] T d616ea7c990c4556a2782e0fde1519a2 P b3941f6d62734fa59b0f1005b814f025 [LEADER]: Connected to new peer: Peer: permanent_uuid: "306a909ee2e344a49f9e7691e2ec8ebe" member_type: VOTER last_known_addr { host: "127.14.147.195" port: 41413 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 22:50:23.537853 17209 consensus_queue.cc:1035] T d616ea7c990c4556a2782e0fde1519a2 P b3941f6d62734fa59b0f1005b814f025 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cb60c60cc23349ab9e59004d1d2983fa" member_type: VOTER last_known_addr { host: "127.14.147.194" port: 39385 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 22:50:23.547603 16648 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:40054:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 22:50:23.547713 16648 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:40054:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 22:50:23.548295 16648 catalog_manager.cc:5782] T 00000000000000000000000000000000 P e2c35b91e0064597ac06279423fc997d: Sending DeleteTablet for 3 replicas of tablet d616ea7c990c4556a2782e0fde1519a2
I20260929 22:50:23.548552 17054 tablet_service.cc:1514] Processing DeleteTablet for tablet d616ea7c990c4556a2782e0fde1519a2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 22:50:23 UTC) from {username='kudu', principal='kudu/127.14.147.254@KRBTEST.COM'} at 127.0.0.1:54066
I20260929 22:50:23.548555 16786 tablet_service.cc:1514] Processing DeleteTablet for tablet d616ea7c990c4556a2782e0fde1519a2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 22:50:23 UTC) from {username='kudu', principal='kudu/127.14.147.254@KRBTEST.COM'} at 127.0.0.1:55110
I20260929 22:50:23.548659 17222 tablet_replica.cc:330] T d616ea7c990c4556a2782e0fde1519a2 P 306a909ee2e344a49f9e7691e2ec8ebe: stopping tablet replica
I20260929 22:50:23.548661 17221 tablet_replica.cc:330] T d616ea7c990c4556a2782e0fde1519a2 P b3941f6d62734fa59b0f1005b814f025: stopping tablet replica
I20260929 22:50:23.548663 16926 tablet_service.cc:1514] Processing DeleteTablet for tablet d616ea7c990c4556a2782e0fde1519a2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 22:50:23 UTC) from {username='kudu', principal='kudu/127.14.147.254@KRBTEST.COM'} at 127.0.0.1:48386
I20260929 22:50:23.548720 17221 raft_consensus.cc:2229] T d616ea7c990c4556a2782e0fde1519a2 P b3941f6d62734fa59b0f1005b814f025 [term 1 LEADER]: Raft consensus shutting down.
I20260929 22:50:23.548718 17222 raft_consensus.cc:2229] T d616ea7c990c4556a2782e0fde1519a2 P 306a909ee2e344a49f9e7691e2ec8ebe [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 22:50:23.548761 17222 raft_consensus.cc:2258] T d616ea7c990c4556a2782e0fde1519a2 P 306a909ee2e344a49f9e7691e2ec8ebe [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 22:50:23.548763 17223 tablet_replica.cc:330] T d616ea7c990c4556a2782e0fde1519a2 P cb60c60cc23349ab9e59004d1d2983fa: stopping tablet replica
I20260929 22:50:23.548774 17221 raft_consensus.cc:2258] T d616ea7c990c4556a2782e0fde1519a2 P b3941f6d62734fa59b0f1005b814f025 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 22:50:23.548811 17223 raft_consensus.cc:2229] T d616ea7c990c4556a2782e0fde1519a2 P cb60c60cc23349ab9e59004d1d2983fa [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 22:50:23.548848 17223 raft_consensus.cc:2258] T d616ea7c990c4556a2782e0fde1519a2 P cb60c60cc23349ab9e59004d1d2983fa [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 22:50:23.549085 17221 ts_tablet_manager.cc:1905] T d616ea7c990c4556a2782e0fde1519a2 P b3941f6d62734fa59b0f1005b814f025: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 22:50:23.549043 16648 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:40054:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260929 22:50:23.549160 17223 ts_tablet_manager.cc:1905] T d616ea7c990c4556a2782e0fde1519a2 P cb60c60cc23349ab9e59004d1d2983fa: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260929 22:50:23.549180 16648 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 22:50:23.549185 17222 ts_tablet_manager.cc:1905] T d616ea7c990c4556a2782e0fde1519a2 P 306a909ee2e344a49f9e7691e2ec8ebe: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 22:50:23.550783 17222 ts_tablet_manager.cc:1918] T d616ea7c990c4556a2782e0fde1519a2 P 306a909ee2e344a49f9e7691e2ec8ebe: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260929 22:50:23.550791 17221 ts_tablet_manager.cc:1918] T d616ea7c990c4556a2782e0fde1519a2 P b3941f6d62734fa59b0f1005b814f025: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260929 22:50:23.550792 17223 ts_tablet_manager.cc:1918] T d616ea7c990c4556a2782e0fde1519a2 P cb60c60cc23349ab9e59004d1d2983fa: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260929 22:50:23.550841 17222 log.cc:1196] T d616ea7c990c4556a2782e0fde1519a2 P 306a909ee2e344a49f9e7691e2ec8ebe: Deleting WAL directory at /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790722213368738-14927-0/minicluster-data/ts-2/wal/wals/d616ea7c990c4556a2782e0fde1519a2
I20260929 22:50:23.550841 17221 log.cc:1196] T d616ea7c990c4556a2782e0fde1519a2 P b3941f6d62734fa59b0f1005b814f025: Deleting WAL directory at /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790722213368738-14927-0/minicluster-data/ts-0/wal/wals/d616ea7c990c4556a2782e0fde1519a2
I20260929 22:50:23.550843 17223 log.cc:1196] T d616ea7c990c4556a2782e0fde1519a2 P cb60c60cc23349ab9e59004d1d2983fa: Deleting WAL directory at /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790722213368738-14927-0/minicluster-data/ts-1/wal/wals/d616ea7c990c4556a2782e0fde1519a2
I20260929 22:50:23.551056 17221 ts_tablet_manager.cc:1939] T d616ea7c990c4556a2782e0fde1519a2 P b3941f6d62734fa59b0f1005b814f025: Deleting consensus metadata
I20260929 22:50:23.551056 17222 ts_tablet_manager.cc:1939] T d616ea7c990c4556a2782e0fde1519a2 P 306a909ee2e344a49f9e7691e2ec8ebe: Deleting consensus metadata
I20260929 22:50:23.551062 17223 ts_tablet_manager.cc:1939] T d616ea7c990c4556a2782e0fde1519a2 P cb60c60cc23349ab9e59004d1d2983fa: Deleting consensus metadata
I20260929 22:50:23.551352 16633 catalog_manager.cc:4841] TS 306a909ee2e344a49f9e7691e2ec8ebe (127.14.147.195:41413): tablet d616ea7c990c4556a2782e0fde1519a2 (table test-table [id=abc1be8a5ce54caf94ca18f938dce24c]) successfully deleted
I20260929 22:50:23.551353 16635 catalog_manager.cc:4841] TS cb60c60cc23349ab9e59004d1d2983fa (127.14.147.194:39385): tablet d616ea7c990c4556a2782e0fde1519a2 (table test-table [id=abc1be8a5ce54caf94ca18f938dce24c]) successfully deleted
I20260929 22:50:23.551596 16633 catalog_manager.cc:4841] TS b3941f6d62734fa59b0f1005b814f025 (127.14.147.193:33765): tablet d616ea7c990c4556a2782e0fde1519a2 (table test-table [id=abc1be8a5ce54caf94ca18f938dce24c]) successfully deleted
I20260929 22:50:23.552006 16786 tablet_service.cc:1467] Processing CreateTablet for tablet 4eae8731a2a8441d92ac9016e385d696 (DEFAULT_TABLE table=test-table [id=c37c925b949e4485bb24bcddadb69005]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 22:50:23.552019 16926 tablet_service.cc:1467] Processing CreateTablet for tablet 4eae8731a2a8441d92ac9016e385d696 (DEFAULT_TABLE table=test-table [id=c37c925b949e4485bb24bcddadb69005]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 22:50:23.552103 16786 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4eae8731a2a8441d92ac9016e385d696. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 22:50:23.552114 16926 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4eae8731a2a8441d92ac9016e385d696. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 22:50:23.552927 17233 tablet_bootstrap.cc:492] T 4eae8731a2a8441d92ac9016e385d696 P cb60c60cc23349ab9e59004d1d2983fa: Bootstrap starting.
I20260929 22:50:23.552973 17054 tablet_service.cc:1467] Processing CreateTablet for tablet 4eae8731a2a8441d92ac9016e385d696 (DEFAULT_TABLE table=test-table [id=c37c925b949e4485bb24bcddadb69005]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 22:50:23.553089 17054 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4eae8731a2a8441d92ac9016e385d696. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 22:50:23.553220 17233 tablet_bootstrap.cc:654] T 4eae8731a2a8441d92ac9016e385d696 P cb60c60cc23349ab9e59004d1d2983fa: Neither blocks nor log segments found. Creating new log.
I20260929 22:50:23.553447 17232 tablet_bootstrap.cc:492] T 4eae8731a2a8441d92ac9016e385d696 P b3941f6d62734fa59b0f1005b814f025: Bootstrap starting.
I20260929 22:50:23.553695 17232 tablet_bootstrap.cc:654] T 4eae8731a2a8441d92ac9016e385d696 P b3941f6d62734fa59b0f1005b814f025: Neither blocks nor log segments found. Creating new log.
I20260929 22:50:23.553798 17233 tablet_bootstrap.cc:492] T 4eae8731a2a8441d92ac9016e385d696 P cb60c60cc23349ab9e59004d1d2983fa: No bootstrap required, opened a new log
I20260929 22:50:23.553838 17233 ts_tablet_manager.cc:1397] T 4eae8731a2a8441d92ac9016e385d696 P cb60c60cc23349ab9e59004d1d2983fa: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260929 22:50:23.554000 17234 tablet_bootstrap.cc:492] T 4eae8731a2a8441d92ac9016e385d696 P 306a909ee2e344a49f9e7691e2ec8ebe: Bootstrap starting.
I20260929 22:50:23.553977 17233 raft_consensus.cc:348] T 4eae8731a2a8441d92ac9016e385d696 P cb60c60cc23349ab9e59004d1d2983fa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3941f6d62734fa59b0f1005b814f025" member_type: VOTER last_known_addr { host: "127.14.147.193" port: 33765 } } peers { permanent_uuid: "cb60c60cc23349ab9e59004d1d2983fa" member_type: VOTER last_known_addr { host: "127.14.147.194" port: 39385 } } peers { permanent_uuid: "306a909ee2e344a49f9e7691e2ec8ebe" member_type: VOTER last_known_addr { host: "127.14.147.195" port: 41413 } }
I20260929 22:50:23.554044 17233 raft_consensus.cc:374] T 4eae8731a2a8441d92ac9016e385d696 P cb60c60cc23349ab9e59004d1d2983fa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 22:50:23.554062 17233 raft_consensus.cc:729] T 4eae8731a2a8441d92ac9016e385d696 P cb60c60cc23349ab9e59004d1d2983fa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cb60c60cc23349ab9e59004d1d2983fa, State: Initialized, Role: FOLLOWER
I20260929 22:50:23.554108 17233 consensus_queue.cc:260] T 4eae8731a2a8441d92ac9016e385d696 P cb60c60cc23349ab9e59004d1d2983fa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3941f6d62734fa59b0f1005b814f025" member_type: VOTER last_known_addr { host: "127.14.147.193" port: 33765 } } peers { permanent_uuid: "cb60c60cc23349ab9e59004d1d2983fa" member_type: VOTER last_known_addr { host: "127.14.147.194" port: 39385 } } peers { permanent_uuid: "306a909ee2e344a49f9e7691e2ec8ebe" member_type: VOTER last_known_addr { host: "127.14.147.195" port: 41413 } }
I20260929 22:50:23.554203 17233 ts_tablet_manager.cc:1428] T 4eae8731a2a8441d92ac9016e385d696 P cb60c60cc23349ab9e59004d1d2983fa: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260929 22:50:23.554276 17234 tablet_bootstrap.cc:654] T 4eae8731a2a8441d92ac9016e385d696 P 306a909ee2e344a49f9e7691e2ec8ebe: Neither blocks nor log segments found. Creating new log.
I20260929 22:50:23.554800 17232 tablet_bootstrap.cc:492] T 4eae8731a2a8441d92ac9016e385d696 P b3941f6d62734fa59b0f1005b814f025: No bootstrap required, opened a new log
I20260929 22:50:23.554848 17234 tablet_bootstrap.cc:492] T 4eae8731a2a8441d92ac9016e385d696 P 306a909ee2e344a49f9e7691e2ec8ebe: No bootstrap required, opened a new log
I20260929 22:50:23.554850 17232 ts_tablet_manager.cc:1397] T 4eae8731a2a8441d92ac9016e385d696 P b3941f6d62734fa59b0f1005b814f025: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260929 22:50:23.554884 17234 ts_tablet_manager.cc:1397] T 4eae8731a2a8441d92ac9016e385d696 P 306a909ee2e344a49f9e7691e2ec8ebe: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260929 22:50:23.554958 17232 raft_consensus.cc:348] T 4eae8731a2a8441d92ac9016e385d696 P b3941f6d62734fa59b0f1005b814f025 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3941f6d62734fa59b0f1005b814f025" member_type: VOTER last_known_addr { host: "127.14.147.193" port: 33765 } } peers { permanent_uuid: "cb60c60cc23349ab9e59004d1d2983fa" member_type: VOTER last_known_addr { host: "127.14.147.194" port: 39385 } } peers { permanent_uuid: "306a909ee2e344a49f9e7691e2ec8ebe" member_type: VOTER last_known_addr { host: "127.14.147.195" port: 41413 } }
I20260929 22:50:23.555020 17232 raft_consensus.cc:374] T 4eae8731a2a8441d92ac9016e385d696 P b3941f6d62734fa59b0f1005b814f025 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 22:50:23.555037 17232 raft_consensus.cc:729] T 4eae8731a2a8441d92ac9016e385d696 P b3941f6d62734fa59b0f1005b814f025 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b3941f6d62734fa59b0f1005b814f025, State: Initialized, Role: FOLLOWER
I20260929 22:50:23.555011 17234 raft_consensus.cc:348] T 4eae8731a2a8441d92ac9016e385d696 P 306a909ee2e344a49f9e7691e2ec8ebe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3941f6d62734fa59b0f1005b814f025" member_type: VOTER last_known_addr { host: "127.14.147.193" port: 33765 } } peers { permanent_uuid: "cb60c60cc23349ab9e59004d1d2983fa" member_type: VOTER last_known_addr { host: "127.14.147.194" port: 39385 } } peers { permanent_uuid: "306a909ee2e344a49f9e7691e2ec8ebe" member_type: VOTER last_known_addr { host: "127.14.147.195" port: 41413 } }
I20260929 22:50:23.555066 17234 raft_consensus.cc:374] T 4eae8731a2a8441d92ac9016e385d696 P 306a909ee2e344a49f9e7691e2ec8ebe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 22:50:23.555084 17234 raft_consensus.cc:729] T 4eae8731a2a8441d92ac9016e385d696 P 306a909ee2e344a49f9e7691e2ec8ebe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 306a909ee2e344a49f9e7691e2ec8ebe, State: Initialized, Role: FOLLOWER
I20260929 22:50:23.555083 17232 consensus_queue.cc:260] T 4eae8731a2a8441d92ac9016e385d696 P b3941f6d62734fa59b0f1005b814f025 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3941f6d62734fa59b0f1005b814f025" member_type: VOTER last_known_addr { host: "127.14.147.193" port: 33765 } } peers { permanent_uuid: "cb60c60cc23349ab9e59004d1d2983fa" member_type: VOTER last_known_addr { host: "127.14.147.194" port: 39385 } } peers { permanent_uuid: "306a909ee2e344a49f9e7691e2ec8ebe" member_type: VOTER last_known_addr { host: "127.14.147.195" port: 41413 } }
I20260929 22:50:23.555124 17234 consensus_queue.cc:260] T 4eae8731a2a8441d92ac9016e385d696 P 306a909ee2e344a49f9e7691e2ec8ebe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3941f6d62734fa59b0f1005b814f025" member_type: VOTER last_known_addr { host: "127.14.147.193" port: 33765 } } peers { permanent_uuid: "cb60c60cc23349ab9e59004d1d2983fa" member_type: VOTER last_known_addr { host: "127.14.147.194" port: 39385 } } peers { permanent_uuid: "306a909ee2e344a49f9e7691e2ec8ebe" member_type: VOTER last_known_addr { host: "127.14.147.195" port: 41413 } }
I20260929 22:50:23.555169 17232 ts_tablet_manager.cc:1428] T 4eae8731a2a8441d92ac9016e385d696 P b3941f6d62734fa59b0f1005b814f025: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260929 22:50:23.555212 17234 ts_tablet_manager.cc:1428] T 4eae8731a2a8441d92ac9016e385d696 P 306a909ee2e344a49f9e7691e2ec8ebe: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260929 22:50:23.590482 17239 raft_consensus.cc:482] T 4eae8731a2a8441d92ac9016e385d696 P cb60c60cc23349ab9e59004d1d2983fa [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 22:50:23.590591 17239 raft_consensus.cc:504] T 4eae8731a2a8441d92ac9016e385d696 P cb60c60cc23349ab9e59004d1d2983fa [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3941f6d62734fa59b0f1005b814f025" member_type: VOTER last_known_addr { host: "127.14.147.193" port: 33765 } } peers { permanent_uuid: "cb60c60cc23349ab9e59004d1d2983fa" member_type: VOTER last_known_addr { host: "127.14.147.194" port: 39385 } } peers { permanent_uuid: "306a909ee2e344a49f9e7691e2ec8ebe" member_type: VOTER last_known_addr { host: "127.14.147.195" port: 41413 } }
I20260929 22:50:23.590915 17239 leader_election.cc:288] T 4eae8731a2a8441d92ac9016e385d696 P cb60c60cc23349ab9e59004d1d2983fa [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b3941f6d62734fa59b0f1005b814f025 (127.14.147.193:33765), 306a909ee2e344a49f9e7691e2ec8ebe (127.14.147.195:41413)
I20260929 22:50:23.594251 17242 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:50:23.591257 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.147.194:40175 (local address 127.14.147.193:33765)
0929 22:50:23.591314 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0929 22:50:23.591317 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 22:50:23.591447 (+ 130us) server_negotiation.cc:407] Connection header received
0929 22:50:23.591563 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 22:50:23.591565 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 22:50:23.591581 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 22:50:23.591636 (+ 55us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 22:50:23.591838 (+ 202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:50:23.592401 (+ 563us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:50:23.593465 (+ 1064us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:50:23.593870 (+ 405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:50:23.593944 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 22:50:23.593959 (+ 15us) server_negotiation.cc:1035] Waiting for connection context
0929 22:50:23.594107 (+ 148us) server_negotiation.cc:299] Negotiation successful
0929 22:50:23.594152 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260929 22:50:23.594292 17249 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:50:23.591086 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.147.193:33765 (local address 127.14.147.194:40175)
0929 22:50:23.591317 (+ 231us) negotiation.cc:107] Waiting for socket to connect
0929 22:50:23.591334 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0929 22:50:23.591468 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 22:50:23.591645 (+ 177us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 22:50:23.591646 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 22:50:23.591650 (+ 4us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 22:50:23.591771 (+ 121us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 22:50:23.591774 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 22:50:23.592514 (+ 740us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 22:50:23.592516 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 22:50:23.593378 (+ 862us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 22:50:23.593380 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 22:50:23.593948 (+ 568us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 22:50:23.593949 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 22:50:23.594029 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 22:50:23.594035 (+ 6us) client_negotiation.cc:769] Sending connection context
0929 22:50:23.594098 (+ 63us) client_negotiation.cc:240] Negotiation successful
0929 22:50:23.594154 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":197,"mutex_wait_us":1,"thread_start_us":84,"threads_started":1}
I20260929 22:50:23.594333 17251 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:50:23.591407 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.147.194:41275 (local address 127.14.147.195:41413)
0929 22:50:23.591537 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0929 22:50:23.591539 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 22:50:23.591551 (+ 12us) server_negotiation.cc:407] Connection header received
0929 22:50:23.591640 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 22:50:23.591641 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 22:50:23.591652 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 22:50:23.591718 (+ 66us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 22:50:23.591895 (+ 177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:50:23.592403 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:50:23.593473 (+ 1070us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:50:23.593909 (+ 436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:50:23.593979 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 22:50:23.593988 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0929 22:50:23.594191 (+ 203us) server_negotiation.cc:299] Negotiation successful
0929 22:50:23.594236 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":112,"thread_start_us":56,"threads_started":1}
I20260929 22:50:23.594605 17250 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:50:23.591290 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.147.195:41413 (local address 127.14.147.194:41275)
0929 22:50:23.591497 (+ 207us) negotiation.cc:107] Waiting for socket to connect
0929 22:50:23.591508 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0929 22:50:23.591583 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 22:50:23.591729 (+ 146us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 22:50:23.591731 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 22:50:23.591733 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 22:50:23.591837 (+ 104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 22:50:23.591839 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 22:50:23.592481 (+ 642us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 22:50:23.592483 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 22:50:23.593385 (+ 902us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 22:50:23.593389 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 22:50:23.593985 (+ 596us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 22:50:23.593986 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 22:50:23.594095 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 22:50:23.594101 (+ 6us) client_negotiation.cc:769] Sending connection context
0929 22:50:23.594183 (+ 82us) client_negotiation.cc:240] Negotiation successful
0929 22:50:23.594243 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":182,"mutex_wait_us":30,"thread_start_us":91,"threads_started":1}
I20260929 22:50:23.594605 16806 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4eae8731a2a8441d92ac9016e385d696" candidate_uuid: "cb60c60cc23349ab9e59004d1d2983fa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b3941f6d62734fa59b0f1005b814f025" is_pre_election: true
I20260929 22:50:23.594758 16806 raft_consensus.cc:2454] T 4eae8731a2a8441d92ac9016e385d696 P b3941f6d62734fa59b0f1005b814f025 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cb60c60cc23349ab9e59004d1d2983fa in term 0.
I20260929 22:50:23.594991 16878 leader_election.cc:302] T 4eae8731a2a8441d92ac9016e385d696 P cb60c60cc23349ab9e59004d1d2983fa [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b3941f6d62734fa59b0f1005b814f025, cb60c60cc23349ab9e59004d1d2983fa; no voters:
I20260929 22:50:23.595032 17086 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4eae8731a2a8441d92ac9016e385d696" candidate_uuid: "cb60c60cc23349ab9e59004d1d2983fa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "306a909ee2e344a49f9e7691e2ec8ebe" is_pre_election: true
I20260929 22:50:23.595110 17086 raft_consensus.cc:2454] T 4eae8731a2a8441d92ac9016e385d696 P 306a909ee2e344a49f9e7691e2ec8ebe [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cb60c60cc23349ab9e59004d1d2983fa in term 0.
I20260929 22:50:23.595187 17239 raft_consensus.cc:2781] T 4eae8731a2a8441d92ac9016e385d696 P cb60c60cc23349ab9e59004d1d2983fa [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 22:50:23.595261 17239 raft_consensus.cc:482] T 4eae8731a2a8441d92ac9016e385d696 P cb60c60cc23349ab9e59004d1d2983fa [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 22:50:23.595287 17239 raft_consensus.cc:3037] T 4eae8731a2a8441d92ac9016e385d696 P cb60c60cc23349ab9e59004d1d2983fa [term 0 FOLLOWER]: Advancing to term 1
I20260929 22:50:23.596261 17239 raft_consensus.cc:504] T 4eae8731a2a8441d92ac9016e385d696 P cb60c60cc23349ab9e59004d1d2983fa [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3941f6d62734fa59b0f1005b814f025" member_type: VOTER last_known_addr { host: "127.14.147.193" port: 33765 } } peers { permanent_uuid: "cb60c60cc23349ab9e59004d1d2983fa" member_type: VOTER last_known_addr { host: "127.14.147.194" port: 39385 } } peers { permanent_uuid: "306a909ee2e344a49f9e7691e2ec8ebe" member_type: VOTER last_known_addr { host: "127.14.147.195" port: 41413 } }
I20260929 22:50:23.596411 17239 leader_election.cc:288] T 4eae8731a2a8441d92ac9016e385d696 P cb60c60cc23349ab9e59004d1d2983fa [CANDIDATE]: Term 1 election: Requested vote from peers b3941f6d62734fa59b0f1005b814f025 (127.14.147.193:33765), 306a909ee2e344a49f9e7691e2ec8ebe (127.14.147.195:41413)
I20260929 22:50:23.596643 17086 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4eae8731a2a8441d92ac9016e385d696" candidate_uuid: "cb60c60cc23349ab9e59004d1d2983fa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "306a909ee2e344a49f9e7691e2ec8ebe"
I20260929 22:50:23.596634 16806 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4eae8731a2a8441d92ac9016e385d696" candidate_uuid: "cb60c60cc23349ab9e59004d1d2983fa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b3941f6d62734fa59b0f1005b814f025"
I20260929 22:50:23.596743 17086 raft_consensus.cc:3037] T 4eae8731a2a8441d92ac9016e385d696 P 306a909ee2e344a49f9e7691e2ec8ebe [term 0 FOLLOWER]: Advancing to term 1
I20260929 22:50:23.596743 16806 raft_consensus.cc:3037] T 4eae8731a2a8441d92ac9016e385d696 P b3941f6d62734fa59b0f1005b814f025 [term 0 FOLLOWER]: Advancing to term 1
I20260929 22:50:23.597596 16806 raft_consensus.cc:2454] T 4eae8731a2a8441d92ac9016e385d696 P b3941f6d62734fa59b0f1005b814f025 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cb60c60cc23349ab9e59004d1d2983fa in term 1.
I20260929 22:50:23.597599 17086 raft_consensus.cc:2454] T 4eae8731a2a8441d92ac9016e385d696 P 306a909ee2e344a49f9e7691e2ec8ebe [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cb60c60cc23349ab9e59004d1d2983fa in term 1.
I20260929 22:50:23.597851 16878 leader_election.cc:302] T 4eae8731a2a8441d92ac9016e385d696 P cb60c60cc23349ab9e59004d1d2983fa [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b3941f6d62734fa59b0f1005b814f025, cb60c60cc23349ab9e59004d1d2983fa; no voters:
I20260929 22:50:23.597975 17239 raft_consensus.cc:2781] T 4eae8731a2a8441d92ac9016e385d696 P cb60c60cc23349ab9e59004d1d2983fa [term 1 FOLLOWER]: Leader election won for term 1
I20260929 22:50:23.598192 17239 raft_consensus.cc:686] T 4eae8731a2a8441d92ac9016e385d696 P cb60c60cc23349ab9e59004d1d2983fa [term 1 LEADER]: Becoming Leader. State: Replica: cb60c60cc23349ab9e59004d1d2983fa, State: Running, Role: LEADER
I20260929 22:50:23.598268 17239 consensus_queue.cc:237] T 4eae8731a2a8441d92ac9016e385d696 P cb60c60cc23349ab9e59004d1d2983fa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3941f6d62734fa59b0f1005b814f025" member_type: VOTER last_known_addr { host: "127.14.147.193" port: 33765 } } peers { permanent_uuid: "cb60c60cc23349ab9e59004d1d2983fa" member_type: VOTER last_known_addr { host: "127.14.147.194" port: 39385 } } peers { permanent_uuid: "306a909ee2e344a49f9e7691e2ec8ebe" member_type: VOTER last_known_addr { host: "127.14.147.195" port: 41413 } }
I20260929 22:50:23.598984 16647 catalog_manager.cc:5495] T 4eae8731a2a8441d92ac9016e385d696 P cb60c60cc23349ab9e59004d1d2983fa reported cstate change: term changed from 0 to 1, leader changed from <none> to cb60c60cc23349ab9e59004d1d2983fa (127.14.147.194). New cstate: current_term: 1 leader_uuid: "cb60c60cc23349ab9e59004d1d2983fa" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3941f6d62734fa59b0f1005b814f025" member_type: VOTER last_known_addr { host: "127.14.147.193" port: 33765 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cb60c60cc23349ab9e59004d1d2983fa" member_type: VOTER last_known_addr { host: "127.14.147.194" port: 39385 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "306a909ee2e344a49f9e7691e2ec8ebe" member_type: VOTER last_known_addr { host: "127.14.147.195" port: 41413 } health_report { overall_health: UNKNOWN } } }
I20260929 22:50:23.605868 17254 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:50:23.603473 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52656 (local address 127.14.147.194:39385)
0929 22:50:23.603648 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0929 22:50:23.603650 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 22:50:23.603657 (+ 7us) server_negotiation.cc:407] Connection header received
0929 22:50:23.603691 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 22:50:23.603692 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 22:50:23.603712 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 22:50:23.603788 (+ 76us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 22:50:23.603882 (+ 94us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:50:23.604389 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:50:23.605152 (+ 763us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:50:23.605379 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:50:23.605442 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 22:50:23.605584 (+ 142us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 22:50:23.605693 (+ 109us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 22:50:23.605745 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0929 22:50:23.605761 (+ 16us) server_negotiation.cc:299] Negotiation successful
0929 22:50:23.605778 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":146,"thread_start_us":120,"threads_started":1}
I20260929 22:50:23.610519 17242 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:50:23.608228 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.147.194:57555 (local address 127.14.147.193:33765)
0929 22:50:23.608299 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0929 22:50:23.608301 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 22:50:23.608350 (+ 49us) server_negotiation.cc:407] Connection header received
0929 22:50:23.608414 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 22:50:23.608415 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 22:50:23.608428 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 22:50:23.608475 (+ 47us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 22:50:23.608596 (+ 121us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:50:23.609084 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:50:23.609828 (+ 744us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:50:23.610052 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:50:23.610107 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 22:50:23.610281 (+ 174us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 22:50:23.610316 (+ 35us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 22:50:23.610356 (+ 40us) server_negotiation.cc:1035] Waiting for connection context
0929 22:50:23.610409 (+ 53us) server_negotiation.cc:299] Negotiation successful
0929 22:50:23.610436 (+ 27us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260929 22:50:23.610554 17257 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:50:23.608142 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.147.193:33765 (local address 127.14.147.194:57555)
0929 22:50:23.608287 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0929 22:50:23.608298 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0929 22:50:23.608357 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 22:50:23.608476 (+ 119us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 22:50:23.608477 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 22:50:23.608479 (+ 2us) client_negotiation.cc:189] Negotiated authn=TOKEN
0929 22:50:23.608541 (+ 62us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 22:50:23.608542 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 22:50:23.609157 (+ 615us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 22:50:23.609158 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 22:50:23.609753 (+ 595us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 22:50:23.609755 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 22:50:23.610131 (+ 376us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 22:50:23.610132 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 22:50:23.610186 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 22:50:23.610208 (+ 22us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0929 22:50:23.610367 (+ 159us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0929 22:50:23.610368 (+ 1us) client_negotiation.cc:769] Sending connection context
0929 22:50:23.610407 (+ 39us) client_negotiation.cc:240] Negotiation successful
0929 22:50:23.610453 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":104,"thread_start_us":79,"threads_started":1}
I20260929 22:50:23.612507 17086 raft_consensus.cc:1261] T 4eae8731a2a8441d92ac9016e385d696 P 306a909ee2e344a49f9e7691e2ec8ebe [term 1 FOLLOWER]: Refusing update from remote peer cb60c60cc23349ab9e59004d1d2983fa: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 22:50:23.612605 16806 raft_consensus.cc:1261] T 4eae8731a2a8441d92ac9016e385d696 P b3941f6d62734fa59b0f1005b814f025 [term 1 FOLLOWER]: Refusing update from remote peer cb60c60cc23349ab9e59004d1d2983fa: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 22:50:23.612746 17239 consensus_queue.cc:1035] T 4eae8731a2a8441d92ac9016e385d696 P cb60c60cc23349ab9e59004d1d2983fa [LEADER]: Connected to new peer: Peer: permanent_uuid: "306a909ee2e344a49f9e7691e2ec8ebe" member_type: VOTER last_known_addr { host: "127.14.147.195" port: 41413 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 22:50:23.612995 17239 consensus_queue.cc:1035] T 4eae8731a2a8441d92ac9016e385d696 P cb60c60cc23349ab9e59004d1d2983fa [LEADER]: Connected to new peer: Peer: permanent_uuid: "b3941f6d62734fa59b0f1005b814f025" member_type: VOTER last_known_addr { host: "127.14.147.193" port: 33765 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 22:50:23.619786 17254 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:50:23.617447 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.147.193:44301 (local address 127.14.147.194:39385)
0929 22:50:23.617542 (+ 95us) server_negotiation.cc:206] Beginning negotiation
0929 22:50:23.617543 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0929 22:50:23.617566 (+ 23us) server_negotiation.cc:407] Connection header received
0929 22:50:23.617626 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 22:50:23.617627 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 22:50:23.617634 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 22:50:23.617669 (+ 35us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 22:50:23.617757 (+ 88us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:50:23.618222 (+ 465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:50:23.619145 (+ 923us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:50:23.619342 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:50:23.619398 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 22:50:23.619611 (+ 213us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 22:50:23.619638 (+ 27us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 22:50:23.619671 (+ 33us) server_negotiation.cc:1035] Waiting for connection context
0929 22:50:23.619709 (+ 38us) server_negotiation.cc:299] Negotiation successful
0929 22:50:23.619727 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260929 22:50:23.619828 17262 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:50:23.617398 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.147.194:39385 (local address 127.14.147.193:44301)
0929 22:50:23.617520 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0929 22:50:23.617531 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0929 22:50:23.617578 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 22:50:23.617672 (+ 94us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 22:50:23.617673 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 22:50:23.617674 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0929 22:50:23.617721 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 22:50:23.617723 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 22:50:23.618301 (+ 578us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 22:50:23.618302 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 22:50:23.619051 (+ 749us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 22:50:23.619054 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 22:50:23.619463 (+ 409us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 22:50:23.619464 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 22:50:23.619542 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 22:50:23.619547 (+ 5us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0929 22:50:23.619678 (+ 131us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0929 22:50:23.619679 (+ 1us) client_negotiation.cc:769] Sending connection context
0929 22:50:23.619706 (+ 27us) client_negotiation.cc:240] Negotiation successful
0929 22:50:23.619763 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":101,"thread_start_us":68,"threads_started":1}
W20260929 22:50:23.620153 17206 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260929 22:50:25.144728 16647 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:40054:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 22:50:25.144829 16647 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:40054:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 22:50:25.145565 16647 catalog_manager.cc:5782] T 00000000000000000000000000000000 P e2c35b91e0064597ac06279423fc997d: Sending DeleteTablet for 3 replicas of tablet 4eae8731a2a8441d92ac9016e385d696
I20260929 22:50:25.145815 16786 tablet_service.cc:1514] Processing DeleteTablet for tablet 4eae8731a2a8441d92ac9016e385d696 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 22:50:25 UTC) from {username='kudu', principal='kudu/127.14.147.254@KRBTEST.COM'} at 127.0.0.1:55110
I20260929 22:50:25.145896 17054 tablet_service.cc:1514] Processing DeleteTablet for tablet 4eae8731a2a8441d92ac9016e385d696 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 22:50:25 UTC) from {username='kudu', principal='kudu/127.14.147.254@KRBTEST.COM'} at 127.0.0.1:54066
I20260929 22:50:25.145992 14927 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhWOCLk/build/release/bin/kudu with pid 16721
I20260929 22:50:25.146417 16926 tablet_service.cc:1514] Processing DeleteTablet for tablet 4eae8731a2a8441d92ac9016e385d696 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 22:50:25 UTC) from {username='kudu', principal='kudu/127.14.147.254@KRBTEST.COM'} at 127.0.0.1:48386
I20260929 22:50:25.147187 17289 tablet_replica.cc:330] T 4eae8731a2a8441d92ac9016e385d696 P cb60c60cc23349ab9e59004d1d2983fa: stopping tablet replica
I20260929 22:50:25.147251 17289 raft_consensus.cc:2229] T 4eae8731a2a8441d92ac9016e385d696 P cb60c60cc23349ab9e59004d1d2983fa [term 1 LEADER]: Raft consensus shutting down.
I20260929 22:50:25.147347 17289 raft_consensus.cc:2258] T 4eae8731a2a8441d92ac9016e385d696 P cb60c60cc23349ab9e59004d1d2983fa [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 22:50:25.147467 17288 tablet_replica.cc:330] T 4eae8731a2a8441d92ac9016e385d696 P 306a909ee2e344a49f9e7691e2ec8ebe: stopping tablet replica
I20260929 22:50:25.147552 17288 raft_consensus.cc:2229] T 4eae8731a2a8441d92ac9016e385d696 P 306a909ee2e344a49f9e7691e2ec8ebe [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 22:50:25.147614 17288 raft_consensus.cc:2258] T 4eae8731a2a8441d92ac9016e385d696 P 306a909ee2e344a49f9e7691e2ec8ebe [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 22:50:25.147670 17289 ts_tablet_manager.cc:1905] T 4eae8731a2a8441d92ac9016e385d696 P cb60c60cc23349ab9e59004d1d2983fa: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 22:50:25.147864 17288 ts_tablet_manager.cc:1905] T 4eae8731a2a8441d92ac9016e385d696 P 306a909ee2e344a49f9e7691e2ec8ebe: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 22:50:25.149268 17289 ts_tablet_manager.cc:1918] T 4eae8731a2a8441d92ac9016e385d696 P cb60c60cc23349ab9e59004d1d2983fa: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260929 22:50:25.149320 17289 log.cc:1196] T 4eae8731a2a8441d92ac9016e385d696 P cb60c60cc23349ab9e59004d1d2983fa: Deleting WAL directory at /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790722213368738-14927-0/minicluster-data/ts-1/wal/wals/4eae8731a2a8441d92ac9016e385d696
I20260929 22:50:25.149430 17288 ts_tablet_manager.cc:1918] T 4eae8731a2a8441d92ac9016e385d696 P 306a909ee2e344a49f9e7691e2ec8ebe: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260929 22:50:25.149485 17288 log.cc:1196] T 4eae8731a2a8441d92ac9016e385d696 P 306a909ee2e344a49f9e7691e2ec8ebe: Deleting WAL directory at /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790722213368738-14927-0/minicluster-data/ts-2/wal/wals/4eae8731a2a8441d92ac9016e385d696
I20260929 22:50:25.149555 17289 ts_tablet_manager.cc:1939] T 4eae8731a2a8441d92ac9016e385d696 P cb60c60cc23349ab9e59004d1d2983fa: Deleting consensus metadata
I20260929 22:50:25.149714 17288 ts_tablet_manager.cc:1939] T 4eae8731a2a8441d92ac9016e385d696 P 306a909ee2e344a49f9e7691e2ec8ebe: Deleting consensus metadata
I20260929 22:50:25.149857 16635 catalog_manager.cc:4841] TS cb60c60cc23349ab9e59004d1d2983fa (127.14.147.194:39385): tablet 4eae8731a2a8441d92ac9016e385d696 (table test-table [id=c37c925b949e4485bb24bcddadb69005]) successfully deleted
I20260929 22:50:25.150008 16633 catalog_manager.cc:4841] TS 306a909ee2e344a49f9e7691e2ec8ebe (127.14.147.195:41413): tablet 4eae8731a2a8441d92ac9016e385d696 (table test-table [id=c37c925b949e4485bb24bcddadb69005]) successfully deleted
W20260929 22:50:25.153158 16633 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.14.147.193:33765 (error 108)
I20260929 22:50:25.153257 14927 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhWOCLk/build/release/bin/kudu with pid 16861
I20260929 22:50:25.158442 14927 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhWOCLk/build/release/bin/kudu with pid 17001
I20260929 22:50:25.159400 17291 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:50:25.158404 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.147.193:33765 (local address 127.0.0.1:36742)
0929 22:50:25.159315 (+ 911us) negotiation.cc:107] Waiting for socket to connect
0929 22:50:25.159361 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.147.193:33765: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":73,"threads_started":1}
W20260929 22:50:25.159556 16633 catalog_manager.cc:4584] TS b3941f6d62734fa59b0f1005b814f025 (127.14.147.193:33765): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 4eae8731a2a8441d92ac9016e385d696: Network error: Client connection negotiation failed: client connection to 127.14.147.193:33765: connect: Connection refused (error 111)
I20260929 22:50:25.162744 14927 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhWOCLk/build/release/bin/kudu with pid 16616
2026-09-29T22:50:25Z chronyd exiting
[ OK ] SecurityITest.TxnSmokeWithDifferentUserTypes (5670 ms)
[ RUN ] SecurityITest.TestNoKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790722213368738-14927-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 22:50:25 dist-test-slave-98mk krb5kdc[17294](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 22:50:25 dist-test-slave-98mk krb5kdc[17294](info): set up 2 sockets
Sep 29 22:50:25 dist-test-slave-98mk krb5kdc[17294](info): commencing operation
krb5kdc: starting...
W20260929 22:50:26.123622 14927 mini_kdc.cc:121] Time spent starting KDC: real 0.941s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790722213368738-14927-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790722213368738-14927-0/krb5kdc/test-user.keytab.
Sep 29 22:50:26 dist-test-slave-98mk krb5kdc[17294](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790722226, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T22:50:26Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T22:50:26Z Disabled control of system clock
WARNING: no policy specified for kudu/127.14.147.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.147.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.147.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790722213368738-14927-0/krb5kdc/kudu_127.14.147.254.keytab.
Entry for principal kudu/127.14.147.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790722213368738-14927-0/krb5kdc/kudu_127.14.147.254.keytab.
I20260929 22:50:26.236126 14927 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhWOCLk/build/release/bin/kudu
/tmp/dist-test-taskhWOCLk/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790722213368738-14927-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790722213368738-14927-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790722213368738-14927-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790722213368738-14927-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.147.254:46675
--webserver_interface=127.14.147.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790722213368738-14927-0/krb5kdc/kudu_127.14.147.254.keytab
--principal=kudu/127.14.147.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.147.212:42265
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.147.254:46675
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790722213368738-14927-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790722213368738-14927-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790722213368738-14927-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 22:50:26.317361 17307 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 22:50:26.317598 17307 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 22:50:26.317623 17307 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 22:50:26.318856 17307 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 22:50:26.318915 17307 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 22:50:26.318929 17307 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 22:50:26.318940 17307 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 22:50:26.318951 17307 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 22:50:26.320621 17307 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.147.212:42265
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790722213368738-14927-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790722213368738-14927-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.14.147.254:46675
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790722213368738-14927-0/krb5kdc/kudu_127.14.147.254.keytab
--principal=kudu/127.14.147.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.147.254:46675
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790722213368738-14927-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.14.147.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17307
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790722213368738-14927-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 21:58:54 UTC on ccycloud.quasar-xguoti.root.comops.site
build id 4681390
I20260929 22:50:26.320854 17307 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 22:50:26.321046 17307 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260929 22:50:26.325896 17307 server_base.cc:1034] running on GCE node
W20260929 22:50:26.325861 17315 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 22:50:26.325884 17313 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 22:50:26.325922 17312 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 22:50:26.326325 17307 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 22:50:26.326628 17307 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 22:50:26.327776 17307 hybrid_clock.cc:648] HybridClock initialized: now 1790722226327758 us; error 33 us; skew 500 ppm
Sep 29 22:50:26 dist-test-slave-98mk krb5kdc[17294](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790722226, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.147.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 22:50:26.330507 17307 init.cc:377] Logged in from keytab as kudu/127.14.147.254@KRBTEST.COM (short username kudu)
I20260929 22:50:26.331001 17307 webserver.cc:460] Webserver started at http://127.14.147.254:41393/ using document root <none> and password file <none>
I20260929 22:50:26.331229 17307 fs_manager.cc:359] Metadata directory not provided
I20260929 22:50:26.331280 17307 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 22:50:26.331406 17307 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 22:50:26.332511 17307 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790722213368738-14927-0/minicluster-data/master-0/data/instance:
uuid: "bb19b9b6b2da4767806da6ebb2093d42"
format_stamp: "Formatted at 2026-09-29 22:50:26 on dist-test-slave-98mk"
server_key: "1f653af5c02269dd47ec04ae3d7b354e"
server_key_iv: "c297f9101686c0368c348e32bfbf19b4"
server_key_version: "encryptionkey@0"
I20260929 22:50:26.332847 17307 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790722213368738-14927-0/minicluster-data/master-0/wal/instance:
uuid: "bb19b9b6b2da4767806da6ebb2093d42"
format_stamp: "Formatted at 2026-09-29 22:50:26 on dist-test-slave-98mk"
server_key: "1f653af5c02269dd47ec04ae3d7b354e"
server_key_iv: "c297f9101686c0368c348e32bfbf19b4"
server_key_version: "encryptionkey@0"
I20260929 22:50:26.334177 17307 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260929 22:50:26.334879 17322 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 22:50:26.335083 17307 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260929 22:50:26.335184 17307 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790722213368738-14927-0/minicluster-data/master-0/data,/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790722213368738-14927-0/minicluster-data/master-0/wal
uuid: "bb19b9b6b2da4767806da6ebb2093d42"
format_stamp: "Formatted at 2026-09-29 22:50:26 on dist-test-slave-98mk"
server_key: "1f653af5c02269dd47ec04ae3d7b354e"
server_key_iv: "c297f9101686c0368c348e32bfbf19b4"
server_key_version: "encryptionkey@0"
I20260929 22:50:26.335253 17307 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790722213368738-14927-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790722213368738-14927-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790722213368738-14927-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 22:50:26.357439 17307 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 22:50:26.359578 17307 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 22:50:26.359730 17307 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 22:50:26.363392 17307 rpc_server.cc:307] RPC server started. Bound to: 127.14.147.254:46675
I20260929 22:50:26.363487 17374 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.147.254:46675 every 8 connection(s)
I20260929 22:50:26.363808 17307 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790722213368738-14927-0/minicluster-data/master-0/data/info.pb
I20260929 22:50:26.364629 17375 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 22:50:26.367056 17375 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bb19b9b6b2da4767806da6ebb2093d42: Bootstrap starting.
I20260929 22:50:26.367682 17375 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P bb19b9b6b2da4767806da6ebb2093d42: Neither blocks nor log segments found. Creating new log.
I20260929 22:50:26.367939 17375 log.cc:824] T 00000000000000000000000000000000 P bb19b9b6b2da4767806da6ebb2093d42: Log is configured to *not* fsync() on all Append() calls
I20260929 22:50:26.368873 17375 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bb19b9b6b2da4767806da6ebb2093d42: No bootstrap required, opened a new log
I20260929 22:50:26.370522 17375 raft_consensus.cc:348] T 00000000000000000000000000000000 P bb19b9b6b2da4767806da6ebb2093d42 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bb19b9b6b2da4767806da6ebb2093d42" member_type: VOTER last_known_addr { host: "127.14.147.254" port: 46675 } }
I20260929 22:50:26.370679 17375 raft_consensus.cc:374] T 00000000000000000000000000000000 P bb19b9b6b2da4767806da6ebb2093d42 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 22:50:26.370713 17375 raft_consensus.cc:729] T 00000000000000000000000000000000 P bb19b9b6b2da4767806da6ebb2093d42 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bb19b9b6b2da4767806da6ebb2093d42, State: Initialized, Role: FOLLOWER
I20260929 22:50:26.370846 17375 consensus_queue.cc:260] T 00000000000000000000000000000000 P bb19b9b6b2da4767806da6ebb2093d42 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bb19b9b6b2da4767806da6ebb2093d42" member_type: VOTER last_known_addr { host: "127.14.147.254" port: 46675 } }
I20260929 22:50:26.370906 17375 raft_consensus.cc:388] T 00000000000000000000000000000000 P bb19b9b6b2da4767806da6ebb2093d42 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 22:50:26.370936 17375 raft_consensus.cc:482] T 00000000000000000000000000000000 P bb19b9b6b2da4767806da6ebb2093d42 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 22:50:26.370983 17375 raft_consensus.cc:3037] T 00000000000000000000000000000000 P bb19b9b6b2da4767806da6ebb2093d42 [term 0 FOLLOWER]: Advancing to term 1
I20260929 22:50:26.371145 14927 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhWOCLk/build/release/bin/kudu as pid 17307
I20260929 22:50:26.371224 14927 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790722213368738-14927-0/minicluster-data/master-0/wal/instance
I20260929 22:50:26.371383 14927 external_mini_cluster.cc:1428] Setting key 354f10dfea0843f76dc62e8417511f64
I20260929 22:50:26.371678 17375 raft_consensus.cc:504] T 00000000000000000000000000000000 P bb19b9b6b2da4767806da6ebb2093d42 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bb19b9b6b2da4767806da6ebb2093d42" member_type: VOTER last_known_addr { host: "127.14.147.254" port: 46675 } }
I20260929 22:50:26.371793 17375 leader_election.cc:302] T 00000000000000000000000000000000 P bb19b9b6b2da4767806da6ebb2093d42 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: bb19b9b6b2da4767806da6ebb2093d42; no voters:
I20260929 22:50:26.372005 17375 leader_election.cc:288] T 00000000000000000000000000000000 P bb19b9b6b2da4767806da6ebb2093d42 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 22:50:26.372035 17379 raft_consensus.cc:2781] T 00000000000000000000000000000000 P bb19b9b6b2da4767806da6ebb2093d42 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 22:50:26.372237 17379 raft_consensus.cc:686] T 00000000000000000000000000000000 P bb19b9b6b2da4767806da6ebb2093d42 [term 1 LEADER]: Becoming Leader. State: Replica: bb19b9b6b2da4767806da6ebb2093d42, State: Running, Role: LEADER
I20260929 22:50:26.372382 17379 consensus_queue.cc:237] T 00000000000000000000000000000000 P bb19b9b6b2da4767806da6ebb2093d42 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bb19b9b6b2da4767806da6ebb2093d42" member_type: VOTER last_known_addr { host: "127.14.147.254" port: 46675 } }
I20260929 22:50:26.372792 17382 sys_catalog.cc:455] T 00000000000000000000000000000000 P bb19b9b6b2da4767806da6ebb2093d42 [sys.catalog]: SysCatalogTable state changed. Reason: New leader bb19b9b6b2da4767806da6ebb2093d42. Latest consensus state: current_term: 1 leader_uuid: "bb19b9b6b2da4767806da6ebb2093d42" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bb19b9b6b2da4767806da6ebb2093d42" member_type: VOTER last_known_addr { host: "127.14.147.254" port: 46675 } } }
I20260929 22:50:26.372800 17381 sys_catalog.cc:455] T 00000000000000000000000000000000 P bb19b9b6b2da4767806da6ebb2093d42 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "bb19b9b6b2da4767806da6ebb2093d42" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bb19b9b6b2da4767806da6ebb2093d42" member_type: VOTER last_known_addr { host: "127.14.147.254" port: 46675 } } }
I20260929 22:50:26.372885 17382 sys_catalog.cc:458] T 00000000000000000000000000000000 P bb19b9b6b2da4767806da6ebb2093d42 [sys.catalog]: This master's current role is: LEADER
I20260929 22:50:26.372891 17381 sys_catalog.cc:458] T 00000000000000000000000000000000 P bb19b9b6b2da4767806da6ebb2093d42 [sys.catalog]: This master's current role is: LEADER
I20260929 22:50:26.373135 17384 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 22:50:26.372386 17375 sys_catalog.cc:564] T 00000000000000000000000000000000 P bb19b9b6b2da4767806da6ebb2093d42 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 22:50:26.373656 17384 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 22:50:26.375206 17384 catalog_manager.cc:1347] Generated new cluster ID: d0cebc9f0a5049069459c87949e8b614
I20260929 22:50:26.375259 17384 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 29 22:50:26 dist-test-slave-98mk krb5kdc[17294](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790722226, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.147.254@KRBTEST.COM
I20260929 22:50:26.382175 17384 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 22:50:26.382673 17384 catalog_manager.cc:1504] Loading token signing keys...
I20260929 22:50:26.384469 17380 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:50:26.371900 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48320 (local address 127.14.147.254:46675)
0929 22:50:26.372124 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0929 22:50:26.372134 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 22:50:26.372166 (+ 32us) server_negotiation.cc:407] Connection header received
0929 22:50:26.372290 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 22:50:26.372298 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 22:50:26.372356 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 22:50:26.372530 (+ 174us) server_negotiation.cc:226] Negotiated authn=SASL
0929 22:50:26.373443 (+ 913us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:50:26.374969 (+ 1526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:50:26.376071 (+ 1102us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:50:26.376689 (+ 618us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:50:26.377119 (+ 430us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 22:50:26.379697 (+ 2578us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 22:50:26.379702 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 22:50:26.379702 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 22:50:26.379716 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0929 22:50:26.382240 (+ 2524us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 22:50:26.382666 (+ 426us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 22:50:26.382669 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 22:50:26.382670 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 22:50:26.382810 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 22:50:26.383023 (+ 213us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 22:50:26.383024 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 22:50:26.383024 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 22:50:26.383376 (+ 352us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 22:50:26.383996 (+ 620us) server_negotiation.cc:1035] Waiting for connection context
0929 22:50:26.384071 (+ 75us) server_negotiation.cc:299] Negotiation successful
0929 22:50:26.384244 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":176,"thread_start_us":102,"threads_started":1}
I20260929 22:50:26.385506 17384 catalog_manager.cc:5868] T 00000000000000000000000000000000 P bb19b9b6b2da4767806da6ebb2093d42: Generated new TSK 0
I20260929 22:50:26.385673 17384 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.14.147.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.147.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.147.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790722213368738-14927-0/krb5kdc/kudu_127.14.147.193.keytab.
Entry for principal kudu/127.14.147.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790722213368738-14927-0/krb5kdc/kudu_127.14.147.193.keytab.
I20260929 22:50:26.410238 14927 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhWOCLk/build/release/bin/kudu
/tmp/dist-test-taskhWOCLk/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790722213368738-14927-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790722213368738-14927-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790722213368738-14927-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790722213368738-14927-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.147.193:0
--local_ip_for_outbound_sockets=127.14.147.193
--webserver_interface=127.14.147.193
--webserver_port=0
--tserver_master_addrs=127.14.147.254:46675
--keytab_file=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790722213368738-14927-0/krb5kdc/kudu_127.14.147.193.keytab
--principal=kudu/127.14.147.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.147.212:42265
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790722213368738-14927-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790722213368738-14927-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790722213368738-14927-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 22:50:26.491231 17401 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 22:50:26.491469 17401 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 22:50:26.491497 17401 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 22:50:26.492784 17401 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 22:50:26.492835 17401 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 22:50:26.492863 17401 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.147.193
I20260929 22:50:26.494392 17401 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.147.212:42265
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790722213368738-14927-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790722213368738-14927-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790722213368738-14927-0/krb5kdc/kudu_127.14.147.193.keytab
--principal=kudu/127.14.147.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.147.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790722213368738-14927-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.14.147.193
--webserver_port=0
--tserver_master_addrs=127.14.147.254:46675
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17401
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.147.193
--log_dir=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790722213368738-14927-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 21:58:54 UTC on ccycloud.quasar-xguoti.root.comops.site
build id 4681390
I20260929 22:50:26.494658 17401 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 22:50:26.494896 17401 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 22:50:26.499828 17407 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 22:50:26.499781 17409 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 22:50:26.499915 17401 server_base.cc:1034] running on GCE node
W20260929 22:50:26.499774 17406 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 22:50:26.500370 17401 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 22:50:26.500609 17401 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 22:50:26.501767 17401 hybrid_clock.cc:648] HybridClock initialized: now 1790722226501753 us; error 29 us; skew 500 ppm
Sep 29 22:50:26 dist-test-slave-98mk krb5kdc[17294](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790722226, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.147.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 22:50:26.504328 17401 init.cc:377] Logged in from keytab as kudu/127.14.147.193@KRBTEST.COM (short username kudu)
I20260929 22:50:26.504801 17401 webserver.cc:460] Webserver started at http://127.14.147.193:39081/ using document root <none> and password file <none>
I20260929 22:50:26.505028 17401 fs_manager.cc:359] Metadata directory not provided
I20260929 22:50:26.505070 17401 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 22:50:26.505185 17401 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 22:50:26.506253 17401 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790722213368738-14927-0/minicluster-data/ts-0/data/instance:
uuid: "69696a4a547b487ba92577684d275e8d"
format_stamp: "Formatted at 2026-09-29 22:50:26 on dist-test-slave-98mk"
server_key: "c5960a73cd1fe97156a6775f9df57c3e"
server_key_iv: "aefeafcb0537f429d7004a3c30b4fff3"
server_key_version: "encryptionkey@0"
I20260929 22:50:26.506577 17401 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790722213368738-14927-0/minicluster-data/ts-0/wal/instance:
uuid: "69696a4a547b487ba92577684d275e8d"
format_stamp: "Formatted at 2026-09-29 22:50:26 on dist-test-slave-98mk"
server_key: "c5960a73cd1fe97156a6775f9df57c3e"
server_key_iv: "aefeafcb0537f429d7004a3c30b4fff3"
server_key_version: "encryptionkey@0"
I20260929 22:50:26.507947 17401 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.002s
I20260929 22:50:26.508701 17416 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 22:50:26.508925 17401 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260929 22:50:26.509021 17401 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790722213368738-14927-0/minicluster-data/ts-0/data,/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790722213368738-14927-0/minicluster-data/ts-0/wal
uuid: "69696a4a547b487ba92577684d275e8d"
format_stamp: "Formatted at 2026-09-29 22:50:26 on dist-test-slave-98mk"
server_key: "c5960a73cd1fe97156a6775f9df57c3e"
server_key_iv: "aefeafcb0537f429d7004a3c30b4fff3"
server_key_version: "encryptionkey@0"
I20260929 22:50:26.509095 17401 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790722213368738-14927-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790722213368738-14927-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790722213368738-14927-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 22:50:26.523797 17401 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 22:50:26.525951 17401 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 22:50:26.526082 17401 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 22:50:26.526340 17401 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 22:50:26.526645 17401 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 22:50:26.526678 17401 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 22:50:26.526700 17401 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 22:50:26.526711 17401 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 22:50:26.532534 17401 rpc_server.cc:307] RPC server started. Bound to: 127.14.147.193:35429
I20260929 22:50:26.532573 17529 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.147.193:35429 every 8 connection(s)
I20260929 22:50:26.532893 17401 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790722213368738-14927-0/minicluster-data/ts-0/data/info.pb
I20260929 22:50:26.535100 14927 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhWOCLk/build/release/bin/kudu as pid 17401
I20260929 22:50:26.535202 14927 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790722213368738-14927-0/minicluster-data/ts-0/wal/instance
I20260929 22:50:26.535384 14927 external_mini_cluster.cc:1428] Setting key efbc2059e735c35b7c8c5d75b7df5614
Sep 29 22:50:26 dist-test-slave-98mk krb5kdc[17294](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790722226, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.147.193@KRBTEST.COM for kudu/127.14.147.254@KRBTEST.COM
I20260929 22:50:26.543376 17380 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:50:26.533575 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.147.193:38739 (local address 127.14.147.254:46675)
0929 22:50:26.533630 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0929 22:50:26.533632 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 22:50:26.533811 (+ 179us) server_negotiation.cc:407] Connection header received
0929 22:50:26.534326 (+ 515us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 22:50:26.534327 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 22:50:26.534340 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 22:50:26.534394 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0929 22:50:26.535629 (+ 1235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:50:26.536230 (+ 601us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:50:26.537615 (+ 1385us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:50:26.537920 (+ 305us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:50:26.538007 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 22:50:26.540471 (+ 2464us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 22:50:26.540473 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 22:50:26.540473 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 22:50:26.540482 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0929 22:50:26.541912 (+ 1430us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 22:50:26.542355 (+ 443us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 22:50:26.542356 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 22:50:26.542356 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 22:50:26.542420 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 22:50:26.542711 (+ 291us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 22:50:26.542711 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 22:50:26.542712 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 22:50:26.542859 (+ 147us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 22:50:26.542933 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0929 22:50:26.543182 (+ 249us) server_negotiation.cc:299] Negotiation successful
0929 22:50:26.543280 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260929 22:50:26.543537 17532 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:50:26.533535 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.147.254:46675 (local address 127.14.147.193:38739)
0929 22:50:26.533682 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0929 22:50:26.533726 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0929 22:50:26.534229 (+ 503us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 22:50:26.534464 (+ 235us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 22:50:26.534468 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 22:50:26.535139 (+ 671us) client_negotiation.cc:189] Negotiated authn=SASL
0929 22:50:26.535483 (+ 344us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 22:50:26.535491 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 22:50:26.536329 (+ 838us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 22:50:26.536330 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 22:50:26.537528 (+ 1198us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 22:50:26.537532 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 22:50:26.538026 (+ 494us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 22:50:26.538027 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 22:50:26.538159 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 22:50:26.538673 (+ 514us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 22:50:26.538681 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0929 22:50:26.540370 (+ 1689us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 22:50:26.542028 (+ 1658us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 22:50:26.542036 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 22:50:26.542038 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 22:50:26.542276 (+ 238us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 22:50:26.542508 (+ 232us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 22:50:26.542509 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 22:50:26.542510 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 22:50:26.542640 (+ 130us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 22:50:26.542921 (+ 281us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 22:50:26.542924 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 22:50:26.543051 (+ 127us) client_negotiation.cc:769] Sending connection context
0929 22:50:26.543133 (+ 82us) client_negotiation.cc:240] Negotiation successful
0929 22:50:26.543267 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":106,"thread_start_us":70,"threads_started":1}
WARNING: no policy specified for kudu/127.14.147.194@KRBTEST.COM; defaulting to no policy
I20260929 22:50:26.544059 17530 heartbeater.cc:347] Connected to a master server at 127.14.147.254:46675
I20260929 22:50:26.544157 17530 heartbeater.cc:464] Registering TS with master...
I20260929 22:50:26.544338 17530 heartbeater.cc:511] Master 127.14.147.254:46675 requested a full tablet report, sending...
I20260929 22:50:26.544862 17339 ts_manager.cc:194] Registered new tserver with Master: 69696a4a547b487ba92577684d275e8d (127.14.147.193:35429)
I20260929 22:50:26.545434 17339 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.147.193@KRBTEST.COM'} at 127.14.147.193:38739
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.147.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.147.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790722213368738-14927-0/krb5kdc/kudu_127.14.147.194.keytab.
Entry for principal kudu/127.14.147.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790722213368738-14927-0/krb5kdc/kudu_127.14.147.194.keytab.
I20260929 22:50:26.558135 14927 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhWOCLk/build/release/bin/kudu
/tmp/dist-test-taskhWOCLk/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790722213368738-14927-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790722213368738-14927-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790722213368738-14927-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790722213368738-14927-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.147.194:0
--local_ip_for_outbound_sockets=127.14.147.194
--webserver_interface=127.14.147.194
--webserver_port=0
--tserver_master_addrs=127.14.147.254:46675
--keytab_file=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790722213368738-14927-0/krb5kdc/kudu_127.14.147.194.keytab
--principal=kudu/127.14.147.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.147.212:42265
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790722213368738-14927-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790722213368738-14927-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790722213368738-14927-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 22:50:26.639709 17535 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 22:50:26.639925 17535 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 22:50:26.639950 17535 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 22:50:26.641179 17535 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 22:50:26.641228 17535 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 22:50:26.641258 17535 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.147.194
I20260929 22:50:26.642781 17535 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.147.212:42265
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790722213368738-14927-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790722213368738-14927-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790722213368738-14927-0/krb5kdc/kudu_127.14.147.194.keytab
--principal=kudu/127.14.147.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.147.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790722213368738-14927-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.14.147.194
--webserver_port=0
--tserver_master_addrs=127.14.147.254:46675
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17535
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.147.194
--log_dir=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790722213368738-14927-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 21:58:54 UTC on ccycloud.quasar-xguoti.root.comops.site
build id 4681390
I20260929 22:50:26.643018 17535 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 22:50:26.643239 17535 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 22:50:26.647936 17541 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 22:50:26.647930 17540 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 22:50:26.648069 17535 server_base.cc:1034] running on GCE node
W20260929 22:50:26.647957 17543 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 22:50:26.648346 17535 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 22:50:26.648588 17535 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 22:50:26.649746 17535 hybrid_clock.cc:648] HybridClock initialized: now 1790722226649729 us; error 34 us; skew 500 ppm
Sep 29 22:50:26 dist-test-slave-98mk krb5kdc[17294](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790722226, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.147.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 22:50:26.652187 17535 init.cc:377] Logged in from keytab as kudu/127.14.147.194@KRBTEST.COM (short username kudu)
I20260929 22:50:26.652618 17535 webserver.cc:460] Webserver started at http://127.14.147.194:38099/ using document root <none> and password file <none>
I20260929 22:50:26.652837 17535 fs_manager.cc:359] Metadata directory not provided
I20260929 22:50:26.652884 17535 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 22:50:26.653008 17535 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 22:50:26.654031 17535 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790722213368738-14927-0/minicluster-data/ts-1/data/instance:
uuid: "57aff16200c9455c8f28627ed0c0fd5f"
format_stamp: "Formatted at 2026-09-29 22:50:26 on dist-test-slave-98mk"
server_key: "fbea9eb486b56ef6439a2e468f0440a4"
server_key_iv: "43eb0ca064cdfd15f41d718cbdee0a06"
server_key_version: "encryptionkey@0"
I20260929 22:50:26.654361 17535 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790722213368738-14927-0/minicluster-data/ts-1/wal/instance:
uuid: "57aff16200c9455c8f28627ed0c0fd5f"
format_stamp: "Formatted at 2026-09-29 22:50:26 on dist-test-slave-98mk"
server_key: "fbea9eb486b56ef6439a2e468f0440a4"
server_key_iv: "43eb0ca064cdfd15f41d718cbdee0a06"
server_key_version: "encryptionkey@0"
I20260929 22:50:26.655639 17535 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260929 22:50:26.656291 17550 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 22:50:26.656438 17535 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260929 22:50:26.656515 17535 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790722213368738-14927-0/minicluster-data/ts-1/data,/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790722213368738-14927-0/minicluster-data/ts-1/wal
uuid: "57aff16200c9455c8f28627ed0c0fd5f"
format_stamp: "Formatted at 2026-09-29 22:50:26 on dist-test-slave-98mk"
server_key: "fbea9eb486b56ef6439a2e468f0440a4"
server_key_iv: "43eb0ca064cdfd15f41d718cbdee0a06"
server_key_version: "encryptionkey@0"
I20260929 22:50:26.656574 17535 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790722213368738-14927-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790722213368738-14927-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790722213368738-14927-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 22:50:26.670815 17535 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 22:50:26.672801 17535 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 22:50:26.672943 17535 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 22:50:26.673161 17535 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 22:50:26.673575 17535 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 22:50:26.673623 17535 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 22:50:26.673655 17535 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 22:50:26.673671 17535 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 22:50:26.679178 17535 rpc_server.cc:307] RPC server started. Bound to: 127.14.147.194:44911
I20260929 22:50:26.679250 17663 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.147.194:44911 every 8 connection(s)
I20260929 22:50:26.679551 17535 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790722213368738-14927-0/minicluster-data/ts-1/data/info.pb
I20260929 22:50:26.683537 14927 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhWOCLk/build/release/bin/kudu as pid 17535
I20260929 22:50:26.683640 14927 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790722213368738-14927-0/minicluster-data/ts-1/wal/instance
I20260929 22:50:26.683806 14927 external_mini_cluster.cc:1428] Setting key d1c0b49eac9f44dc69b0046ca52e6a8e
Sep 29 22:50:26 dist-test-slave-98mk krb5kdc[17294](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790722226, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.147.194@KRBTEST.COM for kudu/127.14.147.254@KRBTEST.COM
I20260929 22:50:26.689334 17380 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:50:26.680211 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.147.194:39647 (local address 127.14.147.254:46675)
0929 22:50:26.680264 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0929 22:50:26.680266 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 22:50:26.680453 (+ 187us) server_negotiation.cc:407] Connection header received
0929 22:50:26.680963 (+ 510us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 22:50:26.680965 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 22:50:26.680979 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 22:50:26.681031 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0929 22:50:26.682022 (+ 991us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:50:26.682548 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:50:26.683813 (+ 1265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:50:26.684133 (+ 320us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:50:26.684244 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 22:50:26.686580 (+ 2336us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 22:50:26.686582 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 22:50:26.686582 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 22:50:26.686592 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0929 22:50:26.687833 (+ 1241us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 22:50:26.688365 (+ 532us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 22:50:26.688366 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 22:50:26.688366 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 22:50:26.688427 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 22:50:26.688709 (+ 282us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 22:50:26.688710 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 22:50:26.688710 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 22:50:26.688821 (+ 111us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 22:50:26.688867 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0929 22:50:26.689193 (+ 326us) server_negotiation.cc:299] Negotiation successful
0929 22:50:26.689263 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260929 22:50:26.689617 17666 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:50:26.680179 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.147.254:46675 (local address 127.14.147.194:39647)
0929 22:50:26.680325 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0929 22:50:26.680365 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0929 22:50:26.680864 (+ 499us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 22:50:26.681102 (+ 238us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 22:50:26.681106 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 22:50:26.681720 (+ 614us) client_negotiation.cc:189] Negotiated authn=SASL
0929 22:50:26.681923 (+ 203us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 22:50:26.681928 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 22:50:26.682637 (+ 709us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 22:50:26.682638 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 22:50:26.683553 (+ 915us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 22:50:26.683556 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 22:50:26.684276 (+ 720us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 22:50:26.684277 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 22:50:26.684433 (+ 156us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 22:50:26.685001 (+ 568us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 22:50:26.685010 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0929 22:50:26.686464 (+ 1454us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 22:50:26.687929 (+ 1465us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 22:50:26.687939 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 22:50:26.687942 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 22:50:26.688259 (+ 317us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 22:50:26.688497 (+ 238us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 22:50:26.688497 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 22:50:26.688498 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 22:50:26.688648 (+ 150us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 22:50:26.688873 (+ 225us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 22:50:26.688879 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 22:50:26.689060 (+ 181us) client_negotiation.cc:769] Sending connection context
0929 22:50:26.689183 (+ 123us) client_negotiation.cc:240] Negotiation successful
0929 22:50:26.689368 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":105,"thread_start_us":73,"threads_started":1}
I20260929 22:50:26.690135 17664 heartbeater.cc:347] Connected to a master server at 127.14.147.254:46675
I20260929 22:50:26.690244 17664 heartbeater.cc:464] Registering TS with master...
I20260929 22:50:26.690449 17664 heartbeater.cc:511] Master 127.14.147.254:46675 requested a full tablet report, sending...
I20260929 22:50:26.690819 17339 ts_manager.cc:194] Registered new tserver with Master: 57aff16200c9455c8f28627ed0c0fd5f (127.14.147.194:44911)
I20260929 22:50:26.691183 17339 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.147.194@KRBTEST.COM'} at 127.14.147.194:39647
WARNING: no policy specified for kudu/127.14.147.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.147.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.147.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790722213368738-14927-0/krb5kdc/kudu_127.14.147.195.keytab.
Entry for principal kudu/127.14.147.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790722213368738-14927-0/krb5kdc/kudu_127.14.147.195.keytab.
I20260929 22:50:26.708523 14927 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhWOCLk/build/release/bin/kudu
/tmp/dist-test-taskhWOCLk/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790722213368738-14927-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790722213368738-14927-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790722213368738-14927-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790722213368738-14927-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.147.195:0
--local_ip_for_outbound_sockets=127.14.147.195
--webserver_interface=127.14.147.195
--webserver_port=0
--tserver_master_addrs=127.14.147.254:46675
--keytab_file=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790722213368738-14927-0/krb5kdc/kudu_127.14.147.195.keytab
--principal=kudu/127.14.147.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.147.212:42265
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790722213368738-14927-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790722213368738-14927-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790722213368738-14927-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 22:50:26.791577 17669 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 22:50:26.791803 17669 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 22:50:26.791828 17669 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 22:50:26.793095 17669 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 22:50:26.793143 17669 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 22:50:26.793171 17669 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.147.195
I20260929 22:50:26.794697 17669 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.147.212:42265
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790722213368738-14927-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790722213368738-14927-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790722213368738-14927-0/krb5kdc/kudu_127.14.147.195.keytab
--principal=kudu/127.14.147.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.147.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790722213368738-14927-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.14.147.195
--webserver_port=0
--tserver_master_addrs=127.14.147.254:46675
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17669
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.147.195
--log_dir=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790722213368738-14927-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 21:58:54 UTC on ccycloud.quasar-xguoti.root.comops.site
build id 4681390
I20260929 22:50:26.794930 17669 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 22:50:26.795131 17669 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 22:50:26.800081 17674 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 22:50:26.800069 17677 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 22:50:26.800140 17669 server_base.cc:1034] running on GCE node
W20260929 22:50:26.800104 17675 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 22:50:26.800474 17669 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 22:50:26.800711 17669 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 22:50:26.801887 17669 hybrid_clock.cc:648] HybridClock initialized: now 1790722226801872 us; error 30 us; skew 500 ppm
Sep 29 22:50:26 dist-test-slave-98mk krb5kdc[17294](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790722226, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.147.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 22:50:26.804356 17669 init.cc:377] Logged in from keytab as kudu/127.14.147.195@KRBTEST.COM (short username kudu)
I20260929 22:50:26.804795 17669 webserver.cc:460] Webserver started at http://127.14.147.195:43905/ using document root <none> and password file <none>
I20260929 22:50:26.804975 17669 fs_manager.cc:359] Metadata directory not provided
I20260929 22:50:26.805012 17669 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 22:50:26.805099 17669 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 22:50:26.806123 17669 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790722213368738-14927-0/minicluster-data/ts-2/data/instance:
uuid: "4f758cab8a7b459a826ea9d24cb04124"
format_stamp: "Formatted at 2026-09-29 22:50:26 on dist-test-slave-98mk"
server_key: "4063f7c74358ffeb1b5a018e215db3d9"
server_key_iv: "1834698d387fb8aad0a83a38621421c7"
server_key_version: "encryptionkey@0"
I20260929 22:50:26.806424 17669 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790722213368738-14927-0/minicluster-data/ts-2/wal/instance:
uuid: "4f758cab8a7b459a826ea9d24cb04124"
format_stamp: "Formatted at 2026-09-29 22:50:26 on dist-test-slave-98mk"
server_key: "4063f7c74358ffeb1b5a018e215db3d9"
server_key_iv: "1834698d387fb8aad0a83a38621421c7"
server_key_version: "encryptionkey@0"
I20260929 22:50:26.807735 17669 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260929 22:50:26.808441 17684 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 22:50:26.808607 17669 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260929 22:50:26.808694 17669 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790722213368738-14927-0/minicluster-data/ts-2/data,/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790722213368738-14927-0/minicluster-data/ts-2/wal
uuid: "4f758cab8a7b459a826ea9d24cb04124"
format_stamp: "Formatted at 2026-09-29 22:50:26 on dist-test-slave-98mk"
server_key: "4063f7c74358ffeb1b5a018e215db3d9"
server_key_iv: "1834698d387fb8aad0a83a38621421c7"
server_key_version: "encryptionkey@0"
I20260929 22:50:26.808766 17669 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790722213368738-14927-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790722213368738-14927-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790722213368738-14927-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 22:50:26.815559 17669 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 22:50:26.817483 17669 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 22:50:26.817605 17669 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 22:50:26.817837 17669 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 22:50:26.818130 17669 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 22:50:26.818163 17669 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 22:50:26.818192 17669 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 22:50:26.818208 17669 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 22:50:26.823921 17669 rpc_server.cc:307] RPC server started. Bound to: 127.14.147.195:38281
I20260929 22:50:26.823966 17797 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.147.195:38281 every 8 connection(s)
I20260929 22:50:26.824290 17669 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790722213368738-14927-0/minicluster-data/ts-2/data/info.pb
Sep 29 22:50:26 dist-test-slave-98mk krb5kdc[17294](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790722226, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.147.195@KRBTEST.COM for kudu/127.14.147.254@KRBTEST.COM
I20260929 22:50:26.833186 17380 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:50:26.825044 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.147.195:52823 (local address 127.14.147.254:46675)
0929 22:50:26.825154 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0929 22:50:26.825157 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 22:50:26.825308 (+ 151us) server_negotiation.cc:407] Connection header received
0929 22:50:26.825822 (+ 514us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 22:50:26.825823 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 22:50:26.825836 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 22:50:26.825889 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0929 22:50:26.826847 (+ 958us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:50:26.827366 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:50:26.828507 (+ 1141us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:50:26.828712 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:50:26.828774 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 22:50:26.830866 (+ 2092us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 22:50:26.830868 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 22:50:26.830868 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 22:50:26.830876 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0929 22:50:26.832036 (+ 1160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 22:50:26.832430 (+ 394us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 22:50:26.832431 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 22:50:26.832431 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 22:50:26.832478 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 22:50:26.832669 (+ 191us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 22:50:26.832669 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 22:50:26.832670 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 22:50:26.832772 (+ 102us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 22:50:26.832816 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0929 22:50:26.833042 (+ 226us) server_negotiation.cc:299] Negotiation successful
0929 22:50:26.833109 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260929 22:50:26.833427 17800 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:50:26.825005 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.147.254:46675 (local address 127.14.147.195:52823)
0929 22:50:26.825175 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0929 22:50:26.825221 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0929 22:50:26.825726 (+ 505us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 22:50:26.825957 (+ 231us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 22:50:26.825960 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 22:50:26.826534 (+ 574us) client_negotiation.cc:189] Negotiated authn=SASL
0929 22:50:26.826748 (+ 214us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 22:50:26.826753 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 22:50:26.827527 (+ 774us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 22:50:26.827529 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 22:50:26.828403 (+ 874us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 22:50:26.828407 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 22:50:26.828786 (+ 379us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 22:50:26.828787 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 22:50:26.828890 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 22:50:26.829401 (+ 511us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 22:50:26.829409 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0929 22:50:26.830770 (+ 1361us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 22:50:26.832116 (+ 1346us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 22:50:26.832124 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 22:50:26.832127 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 22:50:26.832363 (+ 236us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 22:50:26.832523 (+ 160us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 22:50:26.832524 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 22:50:26.832524 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 22:50:26.832625 (+ 101us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 22:50:26.832813 (+ 188us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 22:50:26.832816 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 22:50:26.832944 (+ 128us) client_negotiation.cc:769] Sending connection context
0929 22:50:26.833023 (+ 79us) client_negotiation.cc:240] Negotiation successful
0929 22:50:26.833154 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":79,"threads_started":1}
I20260929 22:50:26.833462 14927 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhWOCLk/build/release/bin/kudu as pid 17669
I20260929 22:50:26.833544 14927 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790722213368738-14927-0/minicluster-data/ts-2/wal/instance
I20260929 22:50:26.833730 14927 external_mini_cluster.cc:1428] Setting key 6a49dded6972d5c131702ba40b7799f3
I20260929 22:50:26.833937 17798 heartbeater.cc:347] Connected to a master server at 127.14.147.254:46675
I20260929 22:50:26.834048 17798 heartbeater.cc:464] Registering TS with master...
I20260929 22:50:26.834235 17798 heartbeater.cc:511] Master 127.14.147.254:46675 requested a full tablet report, sending...
I20260929 22:50:26.834620 17339 ts_manager.cc:194] Registered new tserver with Master: 4f758cab8a7b459a826ea9d24cb04124 (127.14.147.195:38281)
I20260929 22:50:26.834997 17339 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.147.195@KRBTEST.COM'} at 127.14.147.195:52823
I20260929 22:50:26.845427 14927 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260929 22:50:26.857106 17380 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 22:50:26.855658 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48330 (local address 127.14.147.254:46675)
0929 22:50:26.855746 (+ 88us) server_negotiation.cc:206] Beginning negotiation
0929 22:50:26.855749 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 22:50:26.855900 (+ 151us) server_negotiation.cc:407] Connection header received
0929 22:50:26.856036 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 22:50:26.856038 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 22:50:26.856051 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 22:50:26.856103 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0929 22:50:26.857026 (+ 923us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:48330: BlockingRecv error: recv got EOF from 127.0.0.1:48330 (error 108)
Metrics: {"server-negotiator.queue_time_us":47}
I20260929 22:50:26.857937 14927 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhWOCLk/build/release/bin/kudu with pid 17401
I20260929 22:50:26.861929 14927 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhWOCLk/build/release/bin/kudu with pid 17535
I20260929 22:50:26.865783 14927 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhWOCLk/build/release/bin/kudu with pid 17669
I20260929 22:50:26.869588 14927 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhWOCLk/build/release/bin/kudu with pid 17307
2026-09-29T22:50:26Z chronyd exiting
[ OK ] SecurityITest.TestNoKerberosCredentials (1705 ms)
[ RUN ] SecurityITest.SaslPlainFallback
Loading random data
Initializing database '/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790722213368738-14927-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 22:50:26 dist-test-slave-98mk krb5kdc[17810](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 22:50:26 dist-test-slave-98mk krb5kdc[17810](info): set up 2 sockets
Sep 29 22:50:26 dist-test-slave-98mk krb5kdc[17810](info): commencing operation
krb5kdc: starting...
W20260929 22:50:27.823747 14927 mini_kdc.cc:121] Time spent starting KDC: real 0.935s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790722213368738-14927-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790722213368738-14927-0/krb5kdc/test-user.keytab.
Sep 29 22:50:27 dist-test-slave-98mk krb5kdc[17810](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790722227, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T22:50:27Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T22:50:27Z Disabled control of system clock
WARNING: no policy specified for kudu/127.14.147.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.147.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.147.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790722213368738-14927-0/krb5kdc/kudu_127.14.147.254.keytab.
Entry for principal kudu/127.14.147.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790722213368738-14927-0/krb5kdc/kudu_127.14.147.254.keytab.
I20260929 22:50:27.936384 14927 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhWOCLk/build/release/bin/kudu
/tmp/dist-test-taskhWOCLk/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790722213368738-14927-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790722213368738-14927-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790722213368738-14927-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790722213368738-14927-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.147.254:36543
--webserver_interface=127.14.147.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790722213368738-14927-0/krb5kdc/kudu_127.14.147.254.keytab
--principal=kudu/127.14.147.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.147.212:37681
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.147.254:36543
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc-authentication=optional
--user-acl=* with env {KRB5CCNAME=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790722213368738-14927-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790722213368738-14927-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790722213368738-14927-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 22:50:28.018555 17823 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 22:50:28.018782 17823 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 22:50:28.018805 17823 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 22:50:28.020074 17823 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 22:50:28.020128 17823 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 22:50:28.020141 17823 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 22:50:28.020152 17823 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 22:50:28.020164 17823 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 22:50:28.021811 17823 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.147.212:37681
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790722213368738-14927-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790722213368738-14927-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.14.147.254:36543
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790722213368738-14927-0/krb5kdc/kudu_127.14.147.254.keytab
--principal=kudu/127.14.147.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.147.254:36543
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790722213368738-14927-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.14.147.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17823
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790722213368738-14927-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 21:58:54 UTC on ccycloud.quasar-xguoti.root.comops.site
build id 4681390
I20260929 22:50:28.022063 17823 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 22:50:28.022282 17823 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 22:50:28.026993 17829 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 22:50:28.027005 17831 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 22:50:28.027251 17823 server_base.cc:1034] running on GCE node
W20260929 22:50:28.027033 17828 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 22:50:28.027602 17823 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 22:50:28.027868 17823 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 22:50:28.029021 17823 hybrid_clock.cc:648] HybridClock initialized: now 1790722228029002 us; error 33 us; skew 500 ppm
Sep 29 22:50:28 dist-test-slave-98mk krb5kdc[17810](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790722228, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.147.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 22:50:28.032083 17823 init.cc:377] Logged in from keytab as kudu/127.14.147.254@KRBTEST.COM (short username kudu)
I20260929 22:50:28.032585 17823 webserver.cc:460] Webserver started at http://127.14.147.254:42215/ using document root <none> and password file <none>
I20260929 22:50:28.032796 17823 fs_manager.cc:359] Metadata directory not provided
I20260929 22:50:28.032841 17823 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 22:50:28.032956 17823 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 22:50:28.034052 17823 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790722213368738-14927-0/minicluster-data/master-0/data/instance:
uuid: "47d7ef4bfc894bd4b99cfd0b64ff8730"
format_stamp: "Formatted at 2026-09-29 22:50:28 on dist-test-slave-98mk"
server_key: "4346678e6d29288ac009170b752dde87"
server_key_iv: "c59689b16011b1e62f32eac014cd2802"
server_key_version: "encryptionkey@0"
I20260929 22:50:28.034394 17823 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790722213368738-14927-0/minicluster-data/master-0/wal/instance:
uuid: "47d7ef4bfc894bd4b99cfd0b64ff8730"
format_stamp: "Formatted at 2026-09-29 22:50:28 on dist-test-slave-98mk"
server_key: "4346678e6d29288ac009170b752dde87"
server_key_iv: "c59689b16011b1e62f32eac014cd2802"
server_key_version: "encryptionkey@0"
I20260929 22:50:28.035750 17823 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260929 22:50:28.036456 17838 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 22:50:28.036599 17823 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260929 22:50:28.036679 17823 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790722213368738-14927-0/minicluster-data/master-0/data,/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790722213368738-14927-0/minicluster-data/master-0/wal
uuid: "47d7ef4bfc894bd4b99cfd0b64ff8730"
format_stamp: "Formatted at 2026-09-29 22:50:28 on dist-test-slave-98mk"
server_key: "4346678e6d29288ac009170b752dde87"
server_key_iv: "c59689b16011b1e62f32eac014cd2802"
server_key_version: "encryptionkey@0"
I20260929 22:50:28.036748 17823 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790722213368738-14927-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790722213368738-14927-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790722213368738-14927-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 22:50:28.048543 17823 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 22:50:28.051630 17823 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 22:50:28.051764 17823 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 22:50:28.055892 17823 rpc_server.cc:307] RPC server started. Bound to: 127.14.147.254:36543
I20260929 22:50:28.055939 17890 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.147.254:36543 every 8 connection(s)
I20260929 22:50:28.056422 17823 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790722213368738-14927-0/minicluster-data/master-0/data/info.pb
I20260929 22:50:28.057123 17891 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 22:50:28.061259 14927 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhWOCLk/build/release/bin/kudu as pid 17823
I20260929 22:50:28.061375 14927 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790722213368738-14927-0/minicluster-data/master-0/wal/instance
I20260929 22:50:28.061344 17891 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 47d7ef4bfc894bd4b99cfd0b64ff8730: Bootstrap starting.
I20260929 22:50:28.061550 14927 external_mini_cluster.cc:1428] Setting key 696c4da4470302a0ea233d215f07f4ad
I20260929 22:50:28.062431 17891 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 47d7ef4bfc894bd4b99cfd0b64ff8730: Neither blocks nor log segments found. Creating new log.
I20260929 22:50:28.063078 17891 log.cc:824] T 00000000000000000000000000000000 P 47d7ef4bfc894bd4b99cfd0b64ff8730: Log is configured to *not* fsync() on all Append() calls
I20260929 22:50:28.064148 17891 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 47d7ef4bfc894bd4b99cfd0b64ff8730: No bootstrap required, opened a new log
I20260929 22:50:28.066851 17891 raft_consensus.cc:348] T 00000000000000000000000000000000 P 47d7ef4bfc894bd4b99cfd0b64ff8730 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "47d7ef4bfc894bd4b99cfd0b64ff8730" member_type: VOTER last_known_addr { host: "127.14.147.254" port: 36543 } }
I20260929 22:50:28.067073 17891 raft_consensus.cc:374] T 00000000000000000000000000000000 P 47d7ef4bfc894bd4b99cfd0b64ff8730 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 22:50:28.067122 17891 raft_consensus.cc:729] T 00000000000000000000000000000000 P 47d7ef4bfc894bd4b99cfd0b64ff8730 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 47d7ef4bfc894bd4b99cfd0b64ff8730, State: Initialized, Role: FOLLOWER
I20260929 22:50:28.067286 17891 consensus_queue.cc:260] T 00000000000000000000000000000000 P 47d7ef4bfc894bd4b99cfd0b64ff8730 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "47d7ef4bfc894bd4b99cfd0b64ff8730" member_type: VOTER last_known_addr { host: "127.14.147.254" port: 36543 } }
I20260929 22:50:28.067368 17891 raft_consensus.cc:388] T 00000000000000000000000000000000 P 47d7ef4bfc894bd4b99cfd0b64ff8730 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 22:50:28.067396 17891 raft_consensus.cc:482] T 00000000000000000000000000000000 P 47d7ef4bfc894bd4b99cfd0b64ff8730 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 22:50:28.067462 17891 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 47d7ef4bfc894bd4b99cfd0b64ff8730 [term 0 FOLLOWER]: Advancing to term 1
I20260929 22:50:28.068303 17891 raft_consensus.cc:504] T 00000000000000000000000000000000 P 47d7ef4bfc894bd4b99cfd0b64ff8730 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "47d7ef4bfc894bd4b99cfd0b64ff8730" member_type: VOTER last_known_addr { host: "127.14.147.254" port: 36543 } }
I20260929 22:50:28.068414 17891 leader_election.cc:302] T 00000000000000000000000000000000 P 47d7ef4bfc894bd4b99cfd0b64ff8730 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 47d7ef4bfc894bd4b99cfd0b64ff8730; no voters:
I20260929 22:50:28.068601 17891 leader_election.cc:288] T 00000000000000000000000000000000 P 47d7ef4bfc894bd4b99cfd0b64ff8730 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 22:50:28.068662 17896 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 47d7ef4bfc894bd4b99cfd0b64ff8730 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 22:50:28.068835 17896 raft_consensus.cc:686] T 00000000000000000000000000000000 P 47d7ef4bfc894bd4b99cfd0b64ff8730 [term 1 LEADER]: Becoming Leader. State: Replica: 47d7ef4bfc894bd4b99cfd0b64ff8730, State: Running, Role: LEADER
I20260929 22:50:28.068888 17891 sys_catalog.cc:564] T 00000000000000000000000000000000 P 47d7ef4bfc894bd4b99cfd0b64ff8730 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 22:50:28.068979 17896 consensus_queue.cc:237] T 00000000000000000000000000000000 P 47d7ef4bfc894bd4b99cfd0b64ff8730 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "47d7ef4bfc894bd4b99cfd0b64ff8730" member_type: VOTER last_known_addr { host: "127.14.147.254" port: 36543 } }
I20260929 22:50:28.069350 17896 sys_catalog.cc:455] T 00000000000000000000000000000000 P 47d7ef4bfc894bd4b99cfd0b64ff8730 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "47d7ef4bfc894bd4b99cfd0b64ff8730" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "47d7ef4bfc894bd4b99cfd0b64ff8730" member_type: VOTER last_known_addr { host: "127.14.147.254" port: 36543 } } }
I20260929 22:50:28.069433 17896 sys_catalog.cc:458] T 00000000000000000000000000000000 P 47d7ef4bfc894bd4b99cfd0b64ff8730 [sys.catalog]: This master's current role is: LEADER
I20260929 22:50:28.069563 17896 sys_catalog.cc:455] T 00000000000000000000000000000000 P 47d7ef4bfc894bd4b99cfd0b64ff8730 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 47d7ef4bfc894bd4b99cfd0b64ff8730. Latest consensus state: current_term: 1 leader_uuid: "47d7ef4bfc894bd4b99cfd0b64ff8730" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "47d7ef4bfc894bd4b99cfd0b64ff8730" member_type: VOTER last_known_addr { host: "127.14.147.254" port: 36543 } } }
I20260929 22:50:28.069613 17896 sys_catalog.cc:458] T 00000000000000000000000000000000 P 47d7ef4bfc894bd4b99cfd0b64ff8730 [sys.catalog]: This master's current role is: LEADER
Sep 29 22:50:28 dist-test-slave-98mk krb5kdc[17810](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790722227, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.147.254@KRBTEST.COM
W20260929 22:50:28.070921 17911 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 47d7ef4bfc894bd4b99cfd0b64ff8730: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
I20260929 22:50:28.075546 17894 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:50:28.062250 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36534 (local address 127.14.147.254:36543)
0929 22:50:28.062452 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0929 22:50:28.062462 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 22:50:28.062501 (+ 39us) server_negotiation.cc:407] Connection header received
0929 22:50:28.063224 (+ 723us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 22:50:28.063235 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 22:50:28.063310 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 22:50:28.063514 (+ 204us) server_negotiation.cc:226] Negotiated authn=SASL
0929 22:50:28.064228 (+ 714us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:50:28.065786 (+ 1558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:50:28.066937 (+ 1151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:50:28.067621 (+ 684us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:50:28.067724 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 22:50:28.071605 (+ 3881us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 22:50:28.071610 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 22:50:28.071611 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 22:50:28.071625 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0929 22:50:28.073887 (+ 2262us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 22:50:28.074327 (+ 440us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 22:50:28.074330 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 22:50:28.074331 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 22:50:28.074434 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 22:50:28.074637 (+ 203us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 22:50:28.074638 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 22:50:28.074638 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 22:50:28.074905 (+ 267us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 22:50:28.075019 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0929 22:50:28.075152 (+ 133us) server_negotiation.cc:299] Negotiation successful
0929 22:50:28.075274 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":139,"thread_start_us":84,"threads_started":1}
W20260929 22:50:28.075558 17911 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260929 22:50:28.075670 17901 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 22:50:28.075866 17901 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 22:50:28.076488 14927 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 22:50:28.082428 17901 catalog_manager.cc:1347] Generated new cluster ID: f5d69355cab24cf98ed9e1317e6a45f6
I20260929 22:50:28.082499 17901 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 22:50:28.103475 17894 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:50:28.099028 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36544 (local address 127.14.147.254:36543)
0929 22:50:28.099076 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0929 22:50:28.099078 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 22:50:28.099247 (+ 169us) server_negotiation.cc:407] Connection header received
0929 22:50:28.099323 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 22:50:28.099324 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 22:50:28.099350 (+ 26us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 22:50:28.099397 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0929 22:50:28.099854 (+ 457us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:50:28.100635 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:50:28.101647 (+ 1012us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:50:28.101966 (+ 319us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:50:28.102035 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 22:50:28.102826 (+ 791us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 22:50:28.102828 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 22:50:28.102828 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 22:50:28.103105 (+ 277us) server_negotiation.cc:922] calling sasl_server_start()
0929 22:50:28.103129 (+ 24us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 22:50:28.103137 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 22:50:28.103174 (+ 37us) server_negotiation.cc:1035] Waiting for connection context
0929 22:50:28.103315 (+ 141us) server_negotiation.cc:299] Negotiation successful
0929 22:50:28.103350 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260929 22:50:28.106492 17901 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 22:50:28.107232 17901 catalog_manager.cc:1504] Loading token signing keys...
I20260929 22:50:28.111088 17901 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 47d7ef4bfc894bd4b99cfd0b64ff8730: Generated new TSK 0
I20260929 22:50:28.111297 17901 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 22:50:28.115983 14927 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhWOCLk/build/release/bin/kudu with pid 17823
2026-09-29T22:50:28Z chronyd exiting
[ OK ] SecurityITest.SaslPlainFallback (1245 ms)
[ RUN ] SecurityITest.TestUnauthorizedClientKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790722213368738-14927-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 22:50:28 dist-test-slave-98mk krb5kdc[17924](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 22:50:28 dist-test-slave-98mk krb5kdc[17924](info): set up 2 sockets
Sep 29 22:50:28 dist-test-slave-98mk krb5kdc[17924](info): commencing operation
krb5kdc: starting...
W20260929 22:50:29.067796 14927 mini_kdc.cc:121] Time spent starting KDC: real 0.934s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790722213368738-14927-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790722213368738-14927-0/krb5kdc/test-user.keytab.
Sep 29 22:50:29 dist-test-slave-98mk krb5kdc[17924](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790722229, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T22:50:29Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T22:50:29Z Disabled control of system clock
WARNING: no policy specified for kudu/127.14.147.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.147.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.147.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790722213368738-14927-0/krb5kdc/kudu_127.14.147.254.keytab.
Entry for principal kudu/127.14.147.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790722213368738-14927-0/krb5kdc/kudu_127.14.147.254.keytab.
I20260929 22:50:29.176551 14927 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhWOCLk/build/release/bin/kudu
/tmp/dist-test-taskhWOCLk/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790722213368738-14927-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790722213368738-14927-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790722213368738-14927-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790722213368738-14927-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.147.254:39265
--webserver_interface=127.14.147.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790722213368738-14927-0/krb5kdc/kudu_127.14.147.254.keytab
--principal=kudu/127.14.147.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.147.212:35217
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.147.254:39265
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790722213368738-14927-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790722213368738-14927-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790722213368738-14927-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 22:50:29.256919 17937 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 22:50:29.257138 17937 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 22:50:29.257161 17937 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 22:50:29.258323 17937 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 22:50:29.258360 17937 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 22:50:29.258373 17937 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 22:50:29.258384 17937 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 22:50:29.258395 17937 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 22:50:29.259886 17937 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.147.212:35217
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790722213368738-14927-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790722213368738-14927-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.14.147.254:39265
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790722213368738-14927-0/krb5kdc/kudu_127.14.147.254.keytab
--principal=kudu/127.14.147.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.147.254:39265
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790722213368738-14927-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.14.147.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17937
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790722213368738-14927-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 21:58:54 UTC on ccycloud.quasar-xguoti.root.comops.site
build id 4681390
I20260929 22:50:29.260088 17937 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 22:50:29.260289 17937 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 22:50:29.264915 17945 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 22:50:29.264955 17943 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 22:50:29.264941 17942 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 22:50:29.265120 17937 server_base.cc:1034] running on GCE node
I20260929 22:50:29.265341 17937 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 22:50:29.265618 17937 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 22:50:29.266784 17937 hybrid_clock.cc:648] HybridClock initialized: now 1790722229266771 us; error 31 us; skew 500 ppm
Sep 29 22:50:29 dist-test-slave-98mk krb5kdc[17924](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790722229, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.147.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 22:50:29.269232 17937 init.cc:377] Logged in from keytab as kudu/127.14.147.254@KRBTEST.COM (short username kudu)
I20260929 22:50:29.269640 17937 webserver.cc:460] Webserver started at http://127.14.147.254:43325/ using document root <none> and password file <none>
I20260929 22:50:29.269863 17937 fs_manager.cc:359] Metadata directory not provided
I20260929 22:50:29.269912 17937 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 22:50:29.270042 17937 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 22:50:29.271086 17937 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790722213368738-14927-0/minicluster-data/master-0/data/instance:
uuid: "390aabeb03b6468c96015f9c3f4b7e38"
format_stamp: "Formatted at 2026-09-29 22:50:29 on dist-test-slave-98mk"
server_key: "675b98c2244be6d0b2d3926bea40f6f2"
server_key_iv: "628a1728d1fb7548f5c5db6d174bd47c"
server_key_version: "encryptionkey@0"
I20260929 22:50:29.271447 17937 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790722213368738-14927-0/minicluster-data/master-0/wal/instance:
uuid: "390aabeb03b6468c96015f9c3f4b7e38"
format_stamp: "Formatted at 2026-09-29 22:50:29 on dist-test-slave-98mk"
server_key: "675b98c2244be6d0b2d3926bea40f6f2"
server_key_iv: "628a1728d1fb7548f5c5db6d174bd47c"
server_key_version: "encryptionkey@0"
I20260929 22:50:29.272806 17937 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260929 22:50:29.273571 17952 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 22:50:29.273732 17937 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260929 22:50:29.273821 17937 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790722213368738-14927-0/minicluster-data/master-0/data,/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790722213368738-14927-0/minicluster-data/master-0/wal
uuid: "390aabeb03b6468c96015f9c3f4b7e38"
format_stamp: "Formatted at 2026-09-29 22:50:29 on dist-test-slave-98mk"
server_key: "675b98c2244be6d0b2d3926bea40f6f2"
server_key_iv: "628a1728d1fb7548f5c5db6d174bd47c"
server_key_version: "encryptionkey@0"
I20260929 22:50:29.273888 17937 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790722213368738-14927-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790722213368738-14927-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790722213368738-14927-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 22:50:29.284744 17937 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 22:50:29.286895 17937 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 22:50:29.287045 17937 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 22:50:29.291201 17937 rpc_server.cc:307] RPC server started. Bound to: 127.14.147.254:39265
I20260929 22:50:29.291234 18004 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.147.254:39265 every 8 connection(s)
I20260929 22:50:29.291637 17937 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790722213368738-14927-0/minicluster-data/master-0/data/info.pb
I20260929 22:50:29.292629 18005 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 22:50:29.294950 18005 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 390aabeb03b6468c96015f9c3f4b7e38: Bootstrap starting.
I20260929 22:50:29.295574 18005 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 390aabeb03b6468c96015f9c3f4b7e38: Neither blocks nor log segments found. Creating new log.
I20260929 22:50:29.295832 18005 log.cc:824] T 00000000000000000000000000000000 P 390aabeb03b6468c96015f9c3f4b7e38: Log is configured to *not* fsync() on all Append() calls
I20260929 22:50:29.296520 18005 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 390aabeb03b6468c96015f9c3f4b7e38: No bootstrap required, opened a new log
I20260929 22:50:29.298110 18005 raft_consensus.cc:348] T 00000000000000000000000000000000 P 390aabeb03b6468c96015f9c3f4b7e38 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "390aabeb03b6468c96015f9c3f4b7e38" member_type: VOTER last_known_addr { host: "127.14.147.254" port: 39265 } }
I20260929 22:50:29.298259 18005 raft_consensus.cc:374] T 00000000000000000000000000000000 P 390aabeb03b6468c96015f9c3f4b7e38 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 22:50:29.298282 18005 raft_consensus.cc:729] T 00000000000000000000000000000000 P 390aabeb03b6468c96015f9c3f4b7e38 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 390aabeb03b6468c96015f9c3f4b7e38, State: Initialized, Role: FOLLOWER
I20260929 22:50:29.298430 18005 consensus_queue.cc:260] T 00000000000000000000000000000000 P 390aabeb03b6468c96015f9c3f4b7e38 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "390aabeb03b6468c96015f9c3f4b7e38" member_type: VOTER last_known_addr { host: "127.14.147.254" port: 39265 } }
I20260929 22:50:29.298492 18005 raft_consensus.cc:388] T 00000000000000000000000000000000 P 390aabeb03b6468c96015f9c3f4b7e38 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 22:50:29.298514 18005 raft_consensus.cc:482] T 00000000000000000000000000000000 P 390aabeb03b6468c96015f9c3f4b7e38 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 22:50:29.298561 18005 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 390aabeb03b6468c96015f9c3f4b7e38 [term 0 FOLLOWER]: Advancing to term 1
I20260929 22:50:29.299393 18005 raft_consensus.cc:504] T 00000000000000000000000000000000 P 390aabeb03b6468c96015f9c3f4b7e38 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "390aabeb03b6468c96015f9c3f4b7e38" member_type: VOTER last_known_addr { host: "127.14.147.254" port: 39265 } }
I20260929 22:50:29.299530 18005 leader_election.cc:302] T 00000000000000000000000000000000 P 390aabeb03b6468c96015f9c3f4b7e38 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 390aabeb03b6468c96015f9c3f4b7e38; no voters:
I20260929 22:50:29.299691 18005 leader_election.cc:288] T 00000000000000000000000000000000 P 390aabeb03b6468c96015f9c3f4b7e38 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 22:50:29.299723 18008 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 390aabeb03b6468c96015f9c3f4b7e38 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 22:50:29.299880 18008 raft_consensus.cc:686] T 00000000000000000000000000000000 P 390aabeb03b6468c96015f9c3f4b7e38 [term 1 LEADER]: Becoming Leader. State: Replica: 390aabeb03b6468c96015f9c3f4b7e38, State: Running, Role: LEADER
I20260929 22:50:29.299933 18005 sys_catalog.cc:564] T 00000000000000000000000000000000 P 390aabeb03b6468c96015f9c3f4b7e38 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 22:50:29.299971 18008 consensus_queue.cc:237] T 00000000000000000000000000000000 P 390aabeb03b6468c96015f9c3f4b7e38 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "390aabeb03b6468c96015f9c3f4b7e38" member_type: VOTER last_known_addr { host: "127.14.147.254" port: 39265 } }
I20260929 22:50:29.300314 18010 sys_catalog.cc:455] T 00000000000000000000000000000000 P 390aabeb03b6468c96015f9c3f4b7e38 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 390aabeb03b6468c96015f9c3f4b7e38. Latest consensus state: current_term: 1 leader_uuid: "390aabeb03b6468c96015f9c3f4b7e38" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "390aabeb03b6468c96015f9c3f4b7e38" member_type: VOTER last_known_addr { host: "127.14.147.254" port: 39265 } } }
I20260929 22:50:29.300305 18009 sys_catalog.cc:455] T 00000000000000000000000000000000 P 390aabeb03b6468c96015f9c3f4b7e38 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "390aabeb03b6468c96015f9c3f4b7e38" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "390aabeb03b6468c96015f9c3f4b7e38" member_type: VOTER last_known_addr { host: "127.14.147.254" port: 39265 } } }
I20260929 22:50:29.300406 18010 sys_catalog.cc:458] T 00000000000000000000000000000000 P 390aabeb03b6468c96015f9c3f4b7e38 [sys.catalog]: This master's current role is: LEADER
I20260929 22:50:29.300412 18009 sys_catalog.cc:458] T 00000000000000000000000000000000 P 390aabeb03b6468c96015f9c3f4b7e38 [sys.catalog]: This master's current role is: LEADER
I20260929 22:50:29.300603 18015 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 22:50:29.301110 18015 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 22:50:29.301235 14927 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhWOCLk/build/release/bin/kudu as pid 17937
I20260929 22:50:29.301352 14927 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790722213368738-14927-0/minicluster-data/master-0/wal/instance
I20260929 22:50:29.301517 14927 external_mini_cluster.cc:1428] Setting key 4d71b2e80e61ccfa98f9b841c06adcd8
I20260929 22:50:29.302826 18015 catalog_manager.cc:1347] Generated new cluster ID: d6e8f4be20cf4759be0755afaf145058
I20260929 22:50:29.302874 18015 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 29 22:50:29 dist-test-slave-98mk krb5kdc[17924](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790722229, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.147.254@KRBTEST.COM
I20260929 22:50:29.313352 18027 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:50:29.302256 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54110 (local address 127.14.147.254:39265)
0929 22:50:29.302534 (+ 278us) server_negotiation.cc:206] Beginning negotiation
0929 22:50:29.302543 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 22:50:29.302576 (+ 33us) server_negotiation.cc:407] Connection header received
0929 22:50:29.302689 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 22:50:29.302696 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 22:50:29.302751 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 22:50:29.302900 (+ 149us) server_negotiation.cc:226] Negotiated authn=SASL
0929 22:50:29.303659 (+ 759us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:50:29.305068 (+ 1409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:50:29.306162 (+ 1094us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:50:29.306778 (+ 616us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:50:29.306874 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 22:50:29.309649 (+ 2775us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 22:50:29.309655 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 22:50:29.309656 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 22:50:29.309670 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0929 22:50:29.311709 (+ 2039us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 22:50:29.312080 (+ 371us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 22:50:29.312084 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 22:50:29.312084 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 22:50:29.312199 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 22:50:29.312372 (+ 173us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 22:50:29.312373 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 22:50:29.312373 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 22:50:29.312699 (+ 326us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 22:50:29.312880 (+ 181us) server_negotiation.cc:1035] Waiting for connection context
0929 22:50:29.312949 (+ 69us) server_negotiation.cc:299] Negotiation successful
0929 22:50:29.313098 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":234,"thread_start_us":73,"threads_started":1}
I20260929 22:50:29.318236 18015 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 22:50:29.318727 18015 catalog_manager.cc:1504] Loading token signing keys...
I20260929 22:50:29.332854 18015 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 390aabeb03b6468c96015f9c3f4b7e38: Generated new TSK 0
I20260929 22:50:29.333058 18015 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.14.147.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.147.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.147.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790722213368738-14927-0/krb5kdc/kudu_127.14.147.193.keytab.
Entry for principal kudu/127.14.147.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790722213368738-14927-0/krb5kdc/kudu_127.14.147.193.keytab.
I20260929 22:50:29.357849 14927 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhWOCLk/build/release/bin/kudu
/tmp/dist-test-taskhWOCLk/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790722213368738-14927-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790722213368738-14927-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790722213368738-14927-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790722213368738-14927-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.147.193:0
--local_ip_for_outbound_sockets=127.14.147.193
--webserver_interface=127.14.147.193
--webserver_port=0
--tserver_master_addrs=127.14.147.254:39265
--keytab_file=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790722213368738-14927-0/krb5kdc/kudu_127.14.147.193.keytab
--principal=kudu/127.14.147.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.147.212:35217
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790722213368738-14927-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790722213368738-14927-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790722213368738-14927-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 22:50:29.440527 18031 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 22:50:29.440768 18031 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 22:50:29.440794 18031 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 22:50:29.442013 18031 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 22:50:29.442071 18031 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 22:50:29.442109 18031 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.147.193
I20260929 22:50:29.443706 18031 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.147.212:35217
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790722213368738-14927-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790722213368738-14927-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790722213368738-14927-0/krb5kdc/kudu_127.14.147.193.keytab
--principal=kudu/127.14.147.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.147.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790722213368738-14927-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.14.147.193
--webserver_port=0
--tserver_master_addrs=127.14.147.254:39265
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18031
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.147.193
--log_dir=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790722213368738-14927-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 21:58:54 UTC on ccycloud.quasar-xguoti.root.comops.site
build id 4681390
I20260929 22:50:29.443925 18031 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 22:50:29.444120 18031 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 22:50:29.449227 18037 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 22:50:29.449203 18036 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 22:50:29.449268 18031 server_base.cc:1034] running on GCE node
W20260929 22:50:29.449209 18039 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 22:50:29.449661 18031 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 22:50:29.449908 18031 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 22:50:29.451051 18031 hybrid_clock.cc:648] HybridClock initialized: now 1790722229451038 us; error 36 us; skew 500 ppm
Sep 29 22:50:29 dist-test-slave-98mk krb5kdc[17924](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790722229, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.147.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 22:50:29.453651 18031 init.cc:377] Logged in from keytab as kudu/127.14.147.193@KRBTEST.COM (short username kudu)
I20260929 22:50:29.454094 18031 webserver.cc:460] Webserver started at http://127.14.147.193:39043/ using document root <none> and password file <none>
I20260929 22:50:29.454288 18031 fs_manager.cc:359] Metadata directory not provided
I20260929 22:50:29.454329 18031 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 22:50:29.454424 18031 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 22:50:29.455482 18031 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790722213368738-14927-0/minicluster-data/ts-0/data/instance:
uuid: "8b383d8161f448c6aa261642540edf76"
format_stamp: "Formatted at 2026-09-29 22:50:29 on dist-test-slave-98mk"
server_key: "57c65f2532016d0b2c79878780b53e9f"
server_key_iv: "f5e04e4b53d84c4d560894745a6dde13"
server_key_version: "encryptionkey@0"
I20260929 22:50:29.455799 18031 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790722213368738-14927-0/minicluster-data/ts-0/wal/instance:
uuid: "8b383d8161f448c6aa261642540edf76"
format_stamp: "Formatted at 2026-09-29 22:50:29 on dist-test-slave-98mk"
server_key: "57c65f2532016d0b2c79878780b53e9f"
server_key_iv: "f5e04e4b53d84c4d560894745a6dde13"
server_key_version: "encryptionkey@0"
I20260929 22:50:29.456985 18031 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260929 22:50:29.457656 18046 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 22:50:29.457820 18031 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260929 22:50:29.457903 18031 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790722213368738-14927-0/minicluster-data/ts-0/data,/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790722213368738-14927-0/minicluster-data/ts-0/wal
uuid: "8b383d8161f448c6aa261642540edf76"
format_stamp: "Formatted at 2026-09-29 22:50:29 on dist-test-slave-98mk"
server_key: "57c65f2532016d0b2c79878780b53e9f"
server_key_iv: "f5e04e4b53d84c4d560894745a6dde13"
server_key_version: "encryptionkey@0"
I20260929 22:50:29.457962 18031 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790722213368738-14927-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790722213368738-14927-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790722213368738-14927-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 22:50:29.469267 18031 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 22:50:29.471113 18031 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 22:50:29.471222 18031 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 22:50:29.471462 18031 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 22:50:29.471758 18031 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 22:50:29.471793 18031 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 22:50:29.471819 18031 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 22:50:29.471834 18031 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 22:50:29.477463 18031 rpc_server.cc:307] RPC server started. Bound to: 127.14.147.193:36019
I20260929 22:50:29.477600 18159 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.147.193:36019 every 8 connection(s)
I20260929 22:50:29.477852 18031 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790722213368738-14927-0/minicluster-data/ts-0/data/info.pb
I20260929 22:50:29.482774 14927 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhWOCLk/build/release/bin/kudu as pid 18031
I20260929 22:50:29.482875 14927 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790722213368738-14927-0/minicluster-data/ts-0/wal/instance
I20260929 22:50:29.483045 14927 external_mini_cluster.cc:1428] Setting key 7dec750f182b47210653adadaa9f14b5
Sep 29 22:50:29 dist-test-slave-98mk krb5kdc[17924](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790722229, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.147.193@KRBTEST.COM for kudu/127.14.147.254@KRBTEST.COM
I20260929 22:50:29.487911 18027 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:50:29.478506 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.147.193:59001 (local address 127.14.147.254:39265)
0929 22:50:29.478557 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0929 22:50:29.478559 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 22:50:29.478762 (+ 203us) server_negotiation.cc:407] Connection header received
0929 22:50:29.479266 (+ 504us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 22:50:29.479268 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 22:50:29.479284 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 22:50:29.479348 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0929 22:50:29.480346 (+ 998us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:50:29.480875 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:50:29.481927 (+ 1052us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:50:29.482120 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:50:29.482168 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 22:50:29.484466 (+ 2298us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 22:50:29.484470 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 22:50:29.484470 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 22:50:29.484485 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0929 22:50:29.486113 (+ 1628us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 22:50:29.486648 (+ 535us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 22:50:29.486650 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 22:50:29.486650 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 22:50:29.486766 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 22:50:29.487168 (+ 402us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 22:50:29.487169 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 22:50:29.487170 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 22:50:29.487317 (+ 147us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 22:50:29.487381 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0929 22:50:29.487750 (+ 369us) server_negotiation.cc:299] Negotiation successful
0929 22:50:29.487839 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260929 22:50:29.488155 18162 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:50:29.478478 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.147.254:39265 (local address 127.14.147.193:59001)
0929 22:50:29.478633 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0929 22:50:29.478685 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0929 22:50:29.479159 (+ 474us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 22:50:29.479409 (+ 250us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 22:50:29.479439 (+ 30us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 22:50:29.480036 (+ 597us) client_negotiation.cc:189] Negotiated authn=SASL
0929 22:50:29.480238 (+ 202us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 22:50:29.480243 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 22:50:29.480999 (+ 756us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 22:50:29.481000 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 22:50:29.481853 (+ 853us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 22:50:29.481856 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 22:50:29.482180 (+ 324us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 22:50:29.482181 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 22:50:29.482281 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 22:50:29.482823 (+ 542us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 22:50:29.482832 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0929 22:50:29.484323 (+ 1491us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 22:50:29.486208 (+ 1885us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 22:50:29.486216 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 22:50:29.486219 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 22:50:29.486517 (+ 298us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 22:50:29.486922 (+ 405us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 22:50:29.486923 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 22:50:29.486924 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 22:50:29.487064 (+ 140us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 22:50:29.487386 (+ 322us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 22:50:29.487392 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 22:50:29.487604 (+ 212us) client_negotiation.cc:769] Sending connection context
0929 22:50:29.487728 (+ 124us) client_negotiation.cc:240] Negotiation successful
0929 22:50:29.487917 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":72,"threads_started":1}
I20260929 22:50:29.488722 18160 heartbeater.cc:347] Connected to a master server at 127.14.147.254:39265
I20260929 22:50:29.488829 18160 heartbeater.cc:464] Registering TS with master...
I20260929 22:50:29.489034 18160 heartbeater.cc:511] Master 127.14.147.254:39265 requested a full tablet report, sending...
I20260929 22:50:29.489614 17969 ts_manager.cc:194] Registered new tserver with Master: 8b383d8161f448c6aa261642540edf76 (127.14.147.193:36019)
I20260929 22:50:29.490588 17969 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.147.193@KRBTEST.COM'} at 127.14.147.193:59001
WARNING: no policy specified for kudu/127.14.147.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.147.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.147.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790722213368738-14927-0/krb5kdc/kudu_127.14.147.194.keytab.
Entry for principal kudu/127.14.147.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790722213368738-14927-0/krb5kdc/kudu_127.14.147.194.keytab.
I20260929 22:50:29.507722 14927 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhWOCLk/build/release/bin/kudu
/tmp/dist-test-taskhWOCLk/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790722213368738-14927-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790722213368738-14927-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790722213368738-14927-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790722213368738-14927-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.147.194:0
--local_ip_for_outbound_sockets=127.14.147.194
--webserver_interface=127.14.147.194
--webserver_port=0
--tserver_master_addrs=127.14.147.254:39265
--keytab_file=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790722213368738-14927-0/krb5kdc/kudu_127.14.147.194.keytab
--principal=kudu/127.14.147.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.147.212:35217
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790722213368738-14927-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790722213368738-14927-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790722213368738-14927-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 22:50:29.590401 18165 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 22:50:29.590629 18165 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 22:50:29.590673 18165 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 22:50:29.591982 18165 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 22:50:29.592033 18165 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 22:50:29.592072 18165 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.147.194
I20260929 22:50:29.593663 18165 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.147.212:35217
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790722213368738-14927-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790722213368738-14927-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790722213368738-14927-0/krb5kdc/kudu_127.14.147.194.keytab
--principal=kudu/127.14.147.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.147.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790722213368738-14927-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.14.147.194
--webserver_port=0
--tserver_master_addrs=127.14.147.254:39265
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18165
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.147.194
--log_dir=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790722213368738-14927-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 21:58:54 UTC on ccycloud.quasar-xguoti.root.comops.site
build id 4681390
I20260929 22:50:29.593935 18165 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 22:50:29.594153 18165 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 22:50:29.599016 18171 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 22:50:29.599016 18170 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 22:50:29.599020 18173 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 22:50:29.599213 18165 server_base.cc:1034] running on GCE node
I20260929 22:50:29.599474 18165 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 22:50:29.599745 18165 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 22:50:29.600885 18165 hybrid_clock.cc:648] HybridClock initialized: now 1790722229600874 us; error 31 us; skew 500 ppm
Sep 29 22:50:29 dist-test-slave-98mk krb5kdc[17924](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790722229, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.147.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 22:50:29.603482 18165 init.cc:377] Logged in from keytab as kudu/127.14.147.194@KRBTEST.COM (short username kudu)
I20260929 22:50:29.603945 18165 webserver.cc:460] Webserver started at http://127.14.147.194:42239/ using document root <none> and password file <none>
I20260929 22:50:29.604162 18165 fs_manager.cc:359] Metadata directory not provided
I20260929 22:50:29.604209 18165 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 22:50:29.604336 18165 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 22:50:29.605374 18165 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790722213368738-14927-0/minicluster-data/ts-1/data/instance:
uuid: "0c90debafc29459d823a75477476a559"
format_stamp: "Formatted at 2026-09-29 22:50:29 on dist-test-slave-98mk"
server_key: "e216bddefd6ae6e96b8ea405e46db24c"
server_key_iv: "58829b11226f2c563cdc2cd924f4c9c4"
server_key_version: "encryptionkey@0"
I20260929 22:50:29.605712 18165 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790722213368738-14927-0/minicluster-data/ts-1/wal/instance:
uuid: "0c90debafc29459d823a75477476a559"
format_stamp: "Formatted at 2026-09-29 22:50:29 on dist-test-slave-98mk"
server_key: "e216bddefd6ae6e96b8ea405e46db24c"
server_key_iv: "58829b11226f2c563cdc2cd924f4c9c4"
server_key_version: "encryptionkey@0"
I20260929 22:50:29.606980 18165 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260929 22:50:29.607678 18180 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 22:50:29.607818 18165 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260929 22:50:29.607899 18165 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790722213368738-14927-0/minicluster-data/ts-1/data,/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790722213368738-14927-0/minicluster-data/ts-1/wal
uuid: "0c90debafc29459d823a75477476a559"
format_stamp: "Formatted at 2026-09-29 22:50:29 on dist-test-slave-98mk"
server_key: "e216bddefd6ae6e96b8ea405e46db24c"
server_key_iv: "58829b11226f2c563cdc2cd924f4c9c4"
server_key_version: "encryptionkey@0"
I20260929 22:50:29.607961 18165 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790722213368738-14927-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790722213368738-14927-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790722213368738-14927-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 22:50:29.621243 18165 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 22:50:29.623443 18165 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 22:50:29.623602 18165 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 22:50:29.623834 18165 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 22:50:29.624147 18165 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 22:50:29.624181 18165 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 22:50:29.624219 18165 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 22:50:29.624244 18165 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 22:50:29.629802 18165 rpc_server.cc:307] RPC server started. Bound to: 127.14.147.194:42223
I20260929 22:50:29.629851 18294 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.147.194:42223 every 8 connection(s)
I20260929 22:50:29.630182 18165 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790722213368738-14927-0/minicluster-data/ts-1/data/info.pb
I20260929 22:50:29.634507 14927 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhWOCLk/build/release/bin/kudu as pid 18165
I20260929 22:50:29.634619 14927 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790722213368738-14927-0/minicluster-data/ts-1/wal/instance
I20260929 22:50:29.634797 14927 external_mini_cluster.cc:1428] Setting key c83c97f4d740ccc341a48e2fce479866
Sep 29 22:50:29 dist-test-slave-98mk krb5kdc[17924](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790722229, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.147.194@KRBTEST.COM for kudu/127.14.147.254@KRBTEST.COM
I20260929 22:50:29.639565 18027 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:50:29.630888 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.147.194:36649 (local address 127.14.147.254:39265)
0929 22:50:29.630956 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0929 22:50:29.630959 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 22:50:29.631134 (+ 175us) server_negotiation.cc:407] Connection header received
0929 22:50:29.631686 (+ 552us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 22:50:29.631687 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 22:50:29.631699 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 22:50:29.631746 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0929 22:50:29.632660 (+ 914us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:50:29.633151 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:50:29.634104 (+ 953us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:50:29.634296 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:50:29.634351 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 22:50:29.636836 (+ 2485us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 22:50:29.636838 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 22:50:29.636838 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 22:50:29.636845 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0929 22:50:29.638170 (+ 1325us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 22:50:29.638579 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 22:50:29.638580 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 22:50:29.638580 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 22:50:29.638654 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 22:50:29.638873 (+ 219us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 22:50:29.638874 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 22:50:29.638874 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 22:50:29.639008 (+ 134us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 22:50:29.639073 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0929 22:50:29.639363 (+ 290us) server_negotiation.cc:299] Negotiation successful
0929 22:50:29.639467 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260929 22:50:29.639678 18297 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:50:29.630841 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.147.254:39265 (local address 127.14.147.194:36649)
0929 22:50:29.631013 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0929 22:50:29.631056 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0929 22:50:29.631593 (+ 537us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 22:50:29.631825 (+ 232us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 22:50:29.631829 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 22:50:29.632381 (+ 552us) client_negotiation.cc:189] Negotiated authn=SASL
0929 22:50:29.632571 (+ 190us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 22:50:29.632575 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 22:50:29.633232 (+ 657us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 22:50:29.633233 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 22:50:29.634025 (+ 792us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 22:50:29.634027 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 22:50:29.634364 (+ 337us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 22:50:29.634365 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 22:50:29.634496 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 22:50:29.635034 (+ 538us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 22:50:29.635043 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0929 22:50:29.636738 (+ 1695us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 22:50:29.638268 (+ 1530us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 22:50:29.638276 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 22:50:29.638278 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 22:50:29.638508 (+ 230us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 22:50:29.638711 (+ 203us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 22:50:29.638712 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 22:50:29.638712 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 22:50:29.638821 (+ 109us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 22:50:29.639073 (+ 252us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 22:50:29.639078 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 22:50:29.639235 (+ 157us) client_negotiation.cc:769] Sending connection context
0929 22:50:29.639324 (+ 89us) client_negotiation.cc:240] Negotiation successful
0929 22:50:29.639486 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":85,"threads_started":1}
I20260929 22:50:29.640168 18295 heartbeater.cc:347] Connected to a master server at 127.14.147.254:39265
I20260929 22:50:29.640265 18295 heartbeater.cc:464] Registering TS with master...
I20260929 22:50:29.640442 18295 heartbeater.cc:511] Master 127.14.147.254:39265 requested a full tablet report, sending...
I20260929 22:50:29.640816 17969 ts_manager.cc:194] Registered new tserver with Master: 0c90debafc29459d823a75477476a559 (127.14.147.194:42223)
I20260929 22:50:29.641321 17969 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.147.194@KRBTEST.COM'} at 127.14.147.194:36649
WARNING: no policy specified for kudu/127.14.147.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.147.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.147.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790722213368738-14927-0/krb5kdc/kudu_127.14.147.195.keytab.
Entry for principal kudu/127.14.147.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790722213368738-14927-0/krb5kdc/kudu_127.14.147.195.keytab.
I20260929 22:50:29.659014 14927 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhWOCLk/build/release/bin/kudu
/tmp/dist-test-taskhWOCLk/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790722213368738-14927-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790722213368738-14927-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790722213368738-14927-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790722213368738-14927-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.147.195:0
--local_ip_for_outbound_sockets=127.14.147.195
--webserver_interface=127.14.147.195
--webserver_port=0
--tserver_master_addrs=127.14.147.254:39265
--keytab_file=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790722213368738-14927-0/krb5kdc/kudu_127.14.147.195.keytab
--principal=kudu/127.14.147.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.147.212:35217
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790722213368738-14927-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790722213368738-14927-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790722213368738-14927-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 22:50:29.739254 18300 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 22:50:29.739486 18300 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 22:50:29.739509 18300 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 22:50:29.740655 18300 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 22:50:29.740692 18300 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 22:50:29.740717 18300 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.147.195
I20260929 22:50:29.742368 18300 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.147.212:35217
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790722213368738-14927-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790722213368738-14927-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790722213368738-14927-0/krb5kdc/kudu_127.14.147.195.keytab
--principal=kudu/127.14.147.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.147.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790722213368738-14927-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.14.147.195
--webserver_port=0
--tserver_master_addrs=127.14.147.254:39265
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18300
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.147.195
--log_dir=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790722213368738-14927-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 21:58:54 UTC on ccycloud.quasar-xguoti.root.comops.site
build id 4681390
I20260929 22:50:29.742578 18300 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 22:50:29.742772 18300 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 22:50:29.747509 18308 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 22:50:29.747509 18306 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 22:50:29.747539 18305 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 22:50:29.747792 18300 server_base.cc:1034] running on GCE node
I20260929 22:50:29.747989 18300 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 22:50:29.748208 18300 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 22:50:29.749385 18300 hybrid_clock.cc:648] HybridClock initialized: now 1790722229749364 us; error 29 us; skew 500 ppm
Sep 29 22:50:29 dist-test-slave-98mk krb5kdc[17924](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790722229, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.147.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 22:50:29.751929 18300 init.cc:377] Logged in from keytab as kudu/127.14.147.195@KRBTEST.COM (short username kudu)
I20260929 22:50:29.752374 18300 webserver.cc:460] Webserver started at http://127.14.147.195:45981/ using document root <none> and password file <none>
I20260929 22:50:29.752579 18300 fs_manager.cc:359] Metadata directory not provided
I20260929 22:50:29.752628 18300 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 22:50:29.752758 18300 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 22:50:29.753819 18300 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790722213368738-14927-0/minicluster-data/ts-2/data/instance:
uuid: "1e0a167c7b6e4fd5afbb8a37ea84d594"
format_stamp: "Formatted at 2026-09-29 22:50:29 on dist-test-slave-98mk"
server_key: "33b912821c507502f3cb76dfa3a52a44"
server_key_iv: "608ad5d8ef4b3f9ce66a7aa6ee86239d"
server_key_version: "encryptionkey@0"
I20260929 22:50:29.754179 18300 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790722213368738-14927-0/minicluster-data/ts-2/wal/instance:
uuid: "1e0a167c7b6e4fd5afbb8a37ea84d594"
format_stamp: "Formatted at 2026-09-29 22:50:29 on dist-test-slave-98mk"
server_key: "33b912821c507502f3cb76dfa3a52a44"
server_key_iv: "608ad5d8ef4b3f9ce66a7aa6ee86239d"
server_key_version: "encryptionkey@0"
I20260929 22:50:29.755548 18300 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260929 22:50:29.756321 18315 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 22:50:29.756484 18300 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260929 22:50:29.756564 18300 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790722213368738-14927-0/minicluster-data/ts-2/data,/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790722213368738-14927-0/minicluster-data/ts-2/wal
uuid: "1e0a167c7b6e4fd5afbb8a37ea84d594"
format_stamp: "Formatted at 2026-09-29 22:50:29 on dist-test-slave-98mk"
server_key: "33b912821c507502f3cb76dfa3a52a44"
server_key_iv: "608ad5d8ef4b3f9ce66a7aa6ee86239d"
server_key_version: "encryptionkey@0"
I20260929 22:50:29.756635 18300 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790722213368738-14927-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790722213368738-14927-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790722213368738-14927-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 22:50:29.768726 18300 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 22:50:29.770879 18300 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 22:50:29.771021 18300 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 22:50:29.771234 18300 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 22:50:29.771571 18300 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 22:50:29.771616 18300 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 22:50:29.771647 18300 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 22:50:29.771672 18300 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 22:50:29.777300 18300 rpc_server.cc:307] RPC server started. Bound to: 127.14.147.195:40667
I20260929 22:50:29.777351 18428 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.147.195:40667 every 8 connection(s)
I20260929 22:50:29.777681 18300 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790722213368738-14927-0/minicluster-data/ts-2/data/info.pb
Sep 29 22:50:29 dist-test-slave-98mk krb5kdc[17924](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790722229, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.147.195@KRBTEST.COM for kudu/127.14.147.254@KRBTEST.COM
I20260929 22:50:29.784060 14927 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhWOCLk/build/release/bin/kudu as pid 18300
I20260929 22:50:29.784140 14927 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790722213368738-14927-0/minicluster-data/ts-2/wal/instance
I20260929 22:50:29.784337 14927 external_mini_cluster.cc:1428] Setting key 199338a8367a5f28d9e15cf5898f006e
I20260929 22:50:29.786305 18027 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:50:29.778327 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.147.195:37213 (local address 127.14.147.254:39265)
0929 22:50:29.778383 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0929 22:50:29.778385 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 22:50:29.778569 (+ 184us) server_negotiation.cc:407] Connection header received
0929 22:50:29.779068 (+ 499us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 22:50:29.779069 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 22:50:29.779080 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 22:50:29.779133 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0929 22:50:29.780104 (+ 971us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:50:29.780603 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:50:29.781561 (+ 958us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:50:29.781764 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:50:29.781827 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 22:50:29.783825 (+ 1998us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 22:50:29.783826 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 22:50:29.783827 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 22:50:29.783833 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0929 22:50:29.785276 (+ 1443us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 22:50:29.785616 (+ 340us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 22:50:29.785617 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 22:50:29.785617 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 22:50:29.785665 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 22:50:29.785827 (+ 162us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 22:50:29.785827 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 22:50:29.785827 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 22:50:29.785924 (+ 97us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 22:50:29.785968 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0929 22:50:29.786184 (+ 216us) server_negotiation.cc:299] Negotiation successful
0929 22:50:29.786241 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260929 22:50:29.786509 18431 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:50:29.778300 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.147.254:39265 (local address 127.14.147.195:37213)
0929 22:50:29.778462 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0929 22:50:29.778498 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0929 22:50:29.778978 (+ 480us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 22:50:29.779201 (+ 223us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 22:50:29.779205 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 22:50:29.779819 (+ 614us) client_negotiation.cc:189] Negotiated authn=SASL
0929 22:50:29.780017 (+ 198us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 22:50:29.780022 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 22:50:29.780690 (+ 668us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 22:50:29.780691 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 22:50:29.781484 (+ 793us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 22:50:29.781487 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 22:50:29.781846 (+ 359us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 22:50:29.781847 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 22:50:29.781960 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 22:50:29.782437 (+ 477us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 22:50:29.782445 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0929 22:50:29.783735 (+ 1290us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 22:50:29.785342 (+ 1607us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 22:50:29.785348 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 22:50:29.785349 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 22:50:29.785563 (+ 214us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 22:50:29.785702 (+ 139us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 22:50:29.785702 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 22:50:29.785702 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 22:50:29.785788 (+ 86us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 22:50:29.785967 (+ 179us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 22:50:29.785970 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 22:50:29.786082 (+ 112us) client_negotiation.cc:769] Sending connection context
0929 22:50:29.786159 (+ 77us) client_negotiation.cc:240] Negotiation successful
0929 22:50:29.786288 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":82,"threads_started":1}
I20260929 22:50:29.786916 18429 heartbeater.cc:347] Connected to a master server at 127.14.147.254:39265
I20260929 22:50:29.786999 18429 heartbeater.cc:464] Registering TS with master...
I20260929 22:50:29.787171 18429 heartbeater.cc:511] Master 127.14.147.254:39265 requested a full tablet report, sending...
I20260929 22:50:29.787523 17969 ts_manager.cc:194] Registered new tserver with Master: 1e0a167c7b6e4fd5afbb8a37ea84d594 (127.14.147.195:40667)
I20260929 22:50:29.787870 17969 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.147.195@KRBTEST.COM'} at 127.14.147.195:37213
I20260929 22:50:29.796128 14927 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 29 22:50:29 dist-test-slave-98mk krb5kdc[17924](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790722229, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for joe-interloper@KRBTEST.COM:
Sep 29 22:50:29 dist-test-slave-98mk krb5kdc[17924](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790722229, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.14.147.254@KRBTEST.COM
I20260929 22:50:29.819051 18027 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:50:29.812088 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54124 (local address 127.14.147.254:39265)
0929 22:50:29.812149 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0929 22:50:29.812151 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 22:50:29.812240 (+ 89us) server_negotiation.cc:407] Connection header received
0929 22:50:29.812290 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 22:50:29.812291 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 22:50:29.812300 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 22:50:29.812356 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0929 22:50:29.812918 (+ 562us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:50:29.813495 (+ 577us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:50:29.814326 (+ 831us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:50:29.814524 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:50:29.814587 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 22:50:29.816816 (+ 2229us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 22:50:29.816818 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 22:50:29.816818 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 22:50:29.816826 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0929 22:50:29.818026 (+ 1200us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 22:50:29.818399 (+ 373us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 22:50:29.818400 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 22:50:29.818401 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 22:50:29.818461 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 22:50:29.818611 (+ 150us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 22:50:29.818611 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 22:50:29.818611 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 22:50:29.818767 (+ 156us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 22:50:29.818840 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0929 22:50:29.818916 (+ 76us) server_negotiation.cc:299] Negotiation successful
0929 22:50:29.818982 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
W20260929 22:50:29.819296 17969 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.ConnectToMaster from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:54124
I20260929 22:50:29.819759 14927 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhWOCLk/build/release/bin/kudu with pid 18031
I20260929 22:50:29.823634 14927 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhWOCLk/build/release/bin/kudu with pid 18165
I20260929 22:50:29.827338 14927 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhWOCLk/build/release/bin/kudu with pid 18300
I20260929 22:50:29.830917 14927 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhWOCLk/build/release/bin/kudu with pid 17937
2026-09-29T22:50:29Z chronyd exiting
[ OK ] SecurityITest.TestUnauthorizedClientKerberosCredentials (1716 ms)
[ RUN ] SecurityITest.TestAuthorizedSuperuser
Loading random data
Initializing database '/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790722213368738-14927-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 22:50:29 dist-test-slave-98mk krb5kdc[18441](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 22:50:29 dist-test-slave-98mk krb5kdc[18441](info): set up 2 sockets
Sep 29 22:50:29 dist-test-slave-98mk krb5kdc[18441](info): commencing operation
krb5kdc: starting...
W20260929 22:50:30.775871 14927 mini_kdc.cc:121] Time spent starting KDC: real 0.926s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790722213368738-14927-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790722213368738-14927-0/krb5kdc/test-user.keytab.
Sep 29 22:50:30 dist-test-slave-98mk krb5kdc[18441](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790722230, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T22:50:30Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T22:50:30Z Disabled control of system clock
WARNING: no policy specified for kudu/127.14.147.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.147.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.147.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790722213368738-14927-0/krb5kdc/kudu_127.14.147.254.keytab.
Entry for principal kudu/127.14.147.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790722213368738-14927-0/krb5kdc/kudu_127.14.147.254.keytab.
I20260929 22:50:30.887353 14927 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhWOCLk/build/release/bin/kudu
/tmp/dist-test-taskhWOCLk/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790722213368738-14927-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790722213368738-14927-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790722213368738-14927-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790722213368738-14927-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.147.254:44097
--webserver_interface=127.14.147.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790722213368738-14927-0/krb5kdc/kudu_127.14.147.254.keytab
--principal=kudu/127.14.147.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.147.212:41825
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.147.254:44097
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790722213368738-14927-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790722213368738-14927-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790722213368738-14927-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 22:50:30.968194 18454 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 22:50:30.968436 18454 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 22:50:30.968459 18454 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 22:50:30.969702 18454 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 22:50:30.969751 18454 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 22:50:30.969766 18454 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 22:50:30.969779 18454 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 22:50:30.969790 18454 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 22:50:30.971516 18454 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.147.212:41825
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790722213368738-14927-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790722213368738-14927-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.14.147.254:44097
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790722213368738-14927-0/krb5kdc/kudu_127.14.147.254.keytab
--principal=kudu/127.14.147.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.147.254:44097
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790722213368738-14927-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.14.147.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18454
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790722213368738-14927-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 21:58:54 UTC on ccycloud.quasar-xguoti.root.comops.site
build id 4681390
I20260929 22:50:30.971777 18454 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 22:50:30.972004 18454 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 22:50:30.976784 18460 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 22:50:30.976817 18462 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 22:50:30.976788 18459 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 22:50:30.977041 18454 server_base.cc:1034] running on GCE node
I20260929 22:50:30.977298 18454 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 22:50:30.977605 18454 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 22:50:30.978771 18454 hybrid_clock.cc:648] HybridClock initialized: now 1790722230978756 us; error 30 us; skew 500 ppm
Sep 29 22:50:30 dist-test-slave-98mk krb5kdc[18441](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790722230, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.147.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 22:50:30.985970 18454 init.cc:377] Logged in from keytab as kudu/127.14.147.254@KRBTEST.COM (short username kudu)
I20260929 22:50:30.986492 18454 webserver.cc:460] Webserver started at http://127.14.147.254:44321/ using document root <none> and password file <none>
I20260929 22:50:30.986703 18454 fs_manager.cc:359] Metadata directory not provided
I20260929 22:50:30.986752 18454 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 22:50:30.986881 18454 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 22:50:30.988029 18454 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790722213368738-14927-0/minicluster-data/master-0/data/instance:
uuid: "203e02f9c08748228a4ed49da92d5a5b"
format_stamp: "Formatted at 2026-09-29 22:50:30 on dist-test-slave-98mk"
server_key: "571bdd3a72e916fcca42c5dd2d9b4a47"
server_key_iv: "05e18ba0abb1b28788e1167531248a1d"
server_key_version: "encryptionkey@0"
I20260929 22:50:30.988363 18454 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790722213368738-14927-0/minicluster-data/master-0/wal/instance:
uuid: "203e02f9c08748228a4ed49da92d5a5b"
format_stamp: "Formatted at 2026-09-29 22:50:30 on dist-test-slave-98mk"
server_key: "571bdd3a72e916fcca42c5dd2d9b4a47"
server_key_iv: "05e18ba0abb1b28788e1167531248a1d"
server_key_version: "encryptionkey@0"
I20260929 22:50:30.989636 18454 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260929 22:50:30.990302 18469 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 22:50:30.990471 18454 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260929 22:50:30.990556 18454 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790722213368738-14927-0/minicluster-data/master-0/data,/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790722213368738-14927-0/minicluster-data/master-0/wal
uuid: "203e02f9c08748228a4ed49da92d5a5b"
format_stamp: "Formatted at 2026-09-29 22:50:30 on dist-test-slave-98mk"
server_key: "571bdd3a72e916fcca42c5dd2d9b4a47"
server_key_iv: "05e18ba0abb1b28788e1167531248a1d"
server_key_version: "encryptionkey@0"
I20260929 22:50:30.990624 18454 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790722213368738-14927-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790722213368738-14927-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790722213368738-14927-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 22:50:31.005105 18454 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 22:50:31.007269 18454 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 22:50:31.007445 18454 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 22:50:31.011754 18454 rpc_server.cc:307] RPC server started. Bound to: 127.14.147.254:44097
I20260929 22:50:31.011792 18521 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.147.254:44097 every 8 connection(s)
I20260929 22:50:31.012286 18454 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790722213368738-14927-0/minicluster-data/master-0/data/info.pb
I20260929 22:50:31.012912 18522 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 22:50:31.015118 18522 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 203e02f9c08748228a4ed49da92d5a5b: Bootstrap starting.
I20260929 22:50:31.015759 18522 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 203e02f9c08748228a4ed49da92d5a5b: Neither blocks nor log segments found. Creating new log.
I20260929 22:50:31.016001 18522 log.cc:824] T 00000000000000000000000000000000 P 203e02f9c08748228a4ed49da92d5a5b: Log is configured to *not* fsync() on all Append() calls
I20260929 22:50:31.016820 18522 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 203e02f9c08748228a4ed49da92d5a5b: No bootstrap required, opened a new log
I20260929 22:50:31.018388 18522 raft_consensus.cc:348] T 00000000000000000000000000000000 P 203e02f9c08748228a4ed49da92d5a5b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "203e02f9c08748228a4ed49da92d5a5b" member_type: VOTER last_known_addr { host: "127.14.147.254" port: 44097 } }
I20260929 22:50:31.018534 18522 raft_consensus.cc:374] T 00000000000000000000000000000000 P 203e02f9c08748228a4ed49da92d5a5b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 22:50:31.018555 18522 raft_consensus.cc:729] T 00000000000000000000000000000000 P 203e02f9c08748228a4ed49da92d5a5b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 203e02f9c08748228a4ed49da92d5a5b, State: Initialized, Role: FOLLOWER
I20260929 22:50:31.018668 18522 consensus_queue.cc:260] T 00000000000000000000000000000000 P 203e02f9c08748228a4ed49da92d5a5b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "203e02f9c08748228a4ed49da92d5a5b" member_type: VOTER last_known_addr { host: "127.14.147.254" port: 44097 } }
I20260929 22:50:31.018743 18522 raft_consensus.cc:388] T 00000000000000000000000000000000 P 203e02f9c08748228a4ed49da92d5a5b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 22:50:31.018772 18522 raft_consensus.cc:482] T 00000000000000000000000000000000 P 203e02f9c08748228a4ed49da92d5a5b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 22:50:31.018805 18522 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 203e02f9c08748228a4ed49da92d5a5b [term 0 FOLLOWER]: Advancing to term 1
I20260929 22:50:31.019726 18522 raft_consensus.cc:504] T 00000000000000000000000000000000 P 203e02f9c08748228a4ed49da92d5a5b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "203e02f9c08748228a4ed49da92d5a5b" member_type: VOTER last_known_addr { host: "127.14.147.254" port: 44097 } }
I20260929 22:50:31.019814 18522 leader_election.cc:302] T 00000000000000000000000000000000 P 203e02f9c08748228a4ed49da92d5a5b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 203e02f9c08748228a4ed49da92d5a5b; no voters:
I20260929 22:50:31.019935 18522 leader_election.cc:288] T 00000000000000000000000000000000 P 203e02f9c08748228a4ed49da92d5a5b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 22:50:31.020017 18525 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 203e02f9c08748228a4ed49da92d5a5b [term 1 FOLLOWER]: Leader election won for term 1
I20260929 22:50:31.020161 18522 sys_catalog.cc:564] T 00000000000000000000000000000000 P 203e02f9c08748228a4ed49da92d5a5b [sys.catalog]: configured and running, proceeding with master startup.
I20260929 22:50:31.020179 18525 raft_consensus.cc:686] T 00000000000000000000000000000000 P 203e02f9c08748228a4ed49da92d5a5b [term 1 LEADER]: Becoming Leader. State: Replica: 203e02f9c08748228a4ed49da92d5a5b, State: Running, Role: LEADER
I20260929 22:50:31.020267 18525 consensus_queue.cc:237] T 00000000000000000000000000000000 P 203e02f9c08748228a4ed49da92d5a5b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "203e02f9c08748228a4ed49da92d5a5b" member_type: VOTER last_known_addr { host: "127.14.147.254" port: 44097 } }
I20260929 22:50:31.020598 18526 sys_catalog.cc:455] T 00000000000000000000000000000000 P 203e02f9c08748228a4ed49da92d5a5b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "203e02f9c08748228a4ed49da92d5a5b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "203e02f9c08748228a4ed49da92d5a5b" member_type: VOTER last_known_addr { host: "127.14.147.254" port: 44097 } } }
I20260929 22:50:31.020615 18527 sys_catalog.cc:455] T 00000000000000000000000000000000 P 203e02f9c08748228a4ed49da92d5a5b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 203e02f9c08748228a4ed49da92d5a5b. Latest consensus state: current_term: 1 leader_uuid: "203e02f9c08748228a4ed49da92d5a5b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "203e02f9c08748228a4ed49da92d5a5b" member_type: VOTER last_known_addr { host: "127.14.147.254" port: 44097 } } }
I20260929 22:50:31.020699 18526 sys_catalog.cc:458] T 00000000000000000000000000000000 P 203e02f9c08748228a4ed49da92d5a5b [sys.catalog]: This master's current role is: LEADER
I20260929 22:50:31.020701 18527 sys_catalog.cc:458] T 00000000000000000000000000000000 P 203e02f9c08748228a4ed49da92d5a5b [sys.catalog]: This master's current role is: LEADER
I20260929 22:50:31.020921 18534 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 22:50:31.021404 18534 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 22:50:31.022125 14927 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhWOCLk/build/release/bin/kudu as pid 18454
I20260929 22:50:31.022230 14927 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790722213368738-14927-0/minicluster-data/master-0/wal/instance
I20260929 22:50:31.022411 14927 external_mini_cluster.cc:1428] Setting key 7d31f71058c33cd6e068eff707b1606d
I20260929 22:50:31.022903 18534 catalog_manager.cc:1347] Generated new cluster ID: 848d0772d4614b6098eaaa4557cc75e9
I20260929 22:50:31.022953 18534 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 29 22:50:31 dist-test-slave-98mk krb5kdc[18441](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790722230, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.147.254@KRBTEST.COM
I20260929 22:50:31.035166 18545 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:50:31.022876 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44248 (local address 127.14.147.254:44097)
0929 22:50:31.023076 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0929 22:50:31.023085 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 22:50:31.023118 (+ 33us) server_negotiation.cc:407] Connection header received
0929 22:50:31.023231 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 22:50:31.023238 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 22:50:31.023291 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 22:50:31.023440 (+ 149us) server_negotiation.cc:226] Negotiated authn=SASL
0929 22:50:31.024161 (+ 721us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:50:31.025558 (+ 1397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:50:31.026626 (+ 1068us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:50:31.027589 (+ 963us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:50:31.027693 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 22:50:31.031148 (+ 3455us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 22:50:31.031153 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 22:50:31.031154 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 22:50:31.031169 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0929 22:50:31.033461 (+ 2292us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 22:50:31.033872 (+ 411us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 22:50:31.033876 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 22:50:31.033876 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 22:50:31.033996 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 22:50:31.034161 (+ 165us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 22:50:31.034162 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 22:50:31.034162 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 22:50:31.034490 (+ 328us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 22:50:31.034628 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0929 22:50:31.034747 (+ 119us) server_negotiation.cc:299] Negotiation successful
0929 22:50:31.034898 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":156,"thread_start_us":71,"threads_started":1}
I20260929 22:50:31.048418 18534 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 22:50:31.048964 18534 catalog_manager.cc:1504] Loading token signing keys...
I20260929 22:50:31.051661 18534 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 203e02f9c08748228a4ed49da92d5a5b: Generated new TSK 0
I20260929 22:50:31.051819 18534 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.14.147.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.147.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.147.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790722213368738-14927-0/krb5kdc/kudu_127.14.147.193.keytab.
Entry for principal kudu/127.14.147.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790722213368738-14927-0/krb5kdc/kudu_127.14.147.193.keytab.
I20260929 22:50:31.078734 14927 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhWOCLk/build/release/bin/kudu
/tmp/dist-test-taskhWOCLk/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790722213368738-14927-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790722213368738-14927-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790722213368738-14927-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790722213368738-14927-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.147.193:0
--local_ip_for_outbound_sockets=127.14.147.193
--webserver_interface=127.14.147.193
--webserver_port=0
--tserver_master_addrs=127.14.147.254:44097
--keytab_file=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790722213368738-14927-0/krb5kdc/kudu_127.14.147.193.keytab
--principal=kudu/127.14.147.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.147.212:41825
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790722213368738-14927-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790722213368738-14927-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790722213368738-14927-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 22:50:31.161348 18548 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 22:50:31.161581 18548 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 22:50:31.161607 18548 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 22:50:31.163252 18548 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 22:50:31.163456 18548 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 22:50:31.163575 18548 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.147.193
I20260929 22:50:31.166003 18548 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.147.212:41825
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790722213368738-14927-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790722213368738-14927-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790722213368738-14927-0/krb5kdc/kudu_127.14.147.193.keytab
--principal=kudu/127.14.147.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.147.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790722213368738-14927-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.14.147.193
--webserver_port=0
--tserver_master_addrs=127.14.147.254:44097
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18548
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.147.193
--log_dir=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790722213368738-14927-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 21:58:54 UTC on ccycloud.quasar-xguoti.root.comops.site
build id 4681390
I20260929 22:50:31.166272 18548 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 22:50:31.166520 18548 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 22:50:31.171461 18553 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 22:50:31.171463 18554 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 22:50:31.171461 18556 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 22:50:31.171705 18548 server_base.cc:1034] running on GCE node
I20260929 22:50:31.171969 18548 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 22:50:31.172333 18548 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 22:50:31.173491 18548 hybrid_clock.cc:648] HybridClock initialized: now 1790722231173469 us; error 36 us; skew 500 ppm
Sep 29 22:50:31 dist-test-slave-98mk krb5kdc[18441](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790722231, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.147.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 22:50:31.176059 18548 init.cc:377] Logged in from keytab as kudu/127.14.147.193@KRBTEST.COM (short username kudu)
I20260929 22:50:31.176540 18548 webserver.cc:460] Webserver started at http://127.14.147.193:46845/ using document root <none> and password file <none>
I20260929 22:50:31.176745 18548 fs_manager.cc:359] Metadata directory not provided
I20260929 22:50:31.176787 18548 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 22:50:31.176879 18548 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 22:50:31.177989 18548 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790722213368738-14927-0/minicluster-data/ts-0/data/instance:
uuid: "aea1f16241fc492e99fa4042b3af9384"
format_stamp: "Formatted at 2026-09-29 22:50:31 on dist-test-slave-98mk"
server_key: "c4eeea0cf45fbfb3796e830214366cad"
server_key_iv: "2052dde469845a79bc95fd08807bc756"
server_key_version: "encryptionkey@0"
I20260929 22:50:31.178308 18548 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790722213368738-14927-0/minicluster-data/ts-0/wal/instance:
uuid: "aea1f16241fc492e99fa4042b3af9384"
format_stamp: "Formatted at 2026-09-29 22:50:31 on dist-test-slave-98mk"
server_key: "c4eeea0cf45fbfb3796e830214366cad"
server_key_iv: "2052dde469845a79bc95fd08807bc756"
server_key_version: "encryptionkey@0"
I20260929 22:50:31.179616 18548 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260929 22:50:31.180325 18563 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 22:50:31.180459 18548 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260929 22:50:31.180539 18548 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790722213368738-14927-0/minicluster-data/ts-0/data,/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790722213368738-14927-0/minicluster-data/ts-0/wal
uuid: "aea1f16241fc492e99fa4042b3af9384"
format_stamp: "Formatted at 2026-09-29 22:50:31 on dist-test-slave-98mk"
server_key: "c4eeea0cf45fbfb3796e830214366cad"
server_key_iv: "2052dde469845a79bc95fd08807bc756"
server_key_version: "encryptionkey@0"
I20260929 22:50:31.180610 18548 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790722213368738-14927-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790722213368738-14927-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790722213368738-14927-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 22:50:31.192333 18548 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 22:50:31.194447 18548 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 22:50:31.194586 18548 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 22:50:31.194805 18548 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 22:50:31.195088 18548 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 22:50:31.195122 18548 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 22:50:31.195149 18548 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 22:50:31.195174 18548 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 22:50:31.200978 18548 rpc_server.cc:307] RPC server started. Bound to: 127.14.147.193:36245
I20260929 22:50:31.201046 18676 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.147.193:36245 every 8 connection(s)
I20260929 22:50:31.201329 18548 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790722213368738-14927-0/minicluster-data/ts-0/data/info.pb
I20260929 22:50:31.203634 14927 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhWOCLk/build/release/bin/kudu as pid 18548
I20260929 22:50:31.203754 14927 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790722213368738-14927-0/minicluster-data/ts-0/wal/instance
I20260929 22:50:31.203948 14927 external_mini_cluster.cc:1428] Setting key eec4c026de7595995344a9283e1c4687
Sep 29 22:50:31 dist-test-slave-98mk krb5kdc[18441](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790722231, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.147.193@KRBTEST.COM for kudu/127.14.147.254@KRBTEST.COM
I20260929 22:50:31.211072 18545 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:50:31.202025 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.147.193:57067 (local address 127.14.147.254:44097)
0929 22:50:31.202079 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0929 22:50:31.202081 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 22:50:31.202291 (+ 210us) server_negotiation.cc:407] Connection header received
0929 22:50:31.203216 (+ 925us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 22:50:31.203217 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 22:50:31.203232 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 22:50:31.203306 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0929 22:50:31.204166 (+ 860us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:50:31.204773 (+ 607us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:50:31.205790 (+ 1017us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:50:31.206037 (+ 247us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:50:31.206095 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 22:50:31.208434 (+ 2339us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 22:50:31.208435 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 22:50:31.208436 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 22:50:31.208444 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0929 22:50:31.209710 (+ 1266us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 22:50:31.210134 (+ 424us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 22:50:31.210137 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 22:50:31.210137 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 22:50:31.210191 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 22:50:31.210419 (+ 228us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 22:50:31.210419 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 22:50:31.210420 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 22:50:31.210538 (+ 118us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 22:50:31.210595 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0929 22:50:31.210880 (+ 285us) server_negotiation.cc:299] Negotiation successful
0929 22:50:31.210976 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260929 22:50:31.211251 18679 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:50:31.201987 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.147.254:44097 (local address 127.14.147.193:57067)
0929 22:50:31.202167 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0929 22:50:31.202211 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0929 22:50:31.203099 (+ 888us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 22:50:31.203364 (+ 265us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 22:50:31.203368 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 22:50:31.203769 (+ 401us) client_negotiation.cc:189] Negotiated authn=SASL
0929 22:50:31.204057 (+ 288us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 22:50:31.204064 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 22:50:31.204861 (+ 797us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 22:50:31.204863 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 22:50:31.205692 (+ 829us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 22:50:31.205696 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 22:50:31.206113 (+ 417us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 22:50:31.206115 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 22:50:31.206267 (+ 152us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 22:50:31.206799 (+ 532us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 22:50:31.206809 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0929 22:50:31.208336 (+ 1527us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 22:50:31.209802 (+ 1466us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 22:50:31.209811 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 22:50:31.209813 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 22:50:31.210052 (+ 239us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 22:50:31.210247 (+ 195us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 22:50:31.210248 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 22:50:31.210248 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 22:50:31.210362 (+ 114us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 22:50:31.210595 (+ 233us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 22:50:31.210602 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 22:50:31.210748 (+ 146us) client_negotiation.cc:769] Sending connection context
0929 22:50:31.210850 (+ 102us) client_negotiation.cc:240] Negotiation successful
0929 22:50:31.211026 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":137,"thread_start_us":86,"threads_started":1}
WARNING: no policy specified for kudu/127.14.147.194@KRBTEST.COM; defaulting to no policy
I20260929 22:50:31.211798 18677 heartbeater.cc:347] Connected to a master server at 127.14.147.254:44097
I20260929 22:50:31.211889 18677 heartbeater.cc:464] Registering TS with master...
I20260929 22:50:31.212064 18677 heartbeater.cc:511] Master 127.14.147.254:44097 requested a full tablet report, sending...
I20260929 22:50:31.212558 18486 ts_manager.cc:194] Registered new tserver with Master: aea1f16241fc492e99fa4042b3af9384 (127.14.147.193:36245)
I20260929 22:50:31.213201 18486 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.147.193@KRBTEST.COM'} at 127.14.147.193:57067
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.147.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.147.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790722213368738-14927-0/krb5kdc/kudu_127.14.147.194.keytab.
Entry for principal kudu/127.14.147.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790722213368738-14927-0/krb5kdc/kudu_127.14.147.194.keytab.
I20260929 22:50:31.228380 14927 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhWOCLk/build/release/bin/kudu
/tmp/dist-test-taskhWOCLk/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790722213368738-14927-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790722213368738-14927-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790722213368738-14927-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790722213368738-14927-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.147.194:0
--local_ip_for_outbound_sockets=127.14.147.194
--webserver_interface=127.14.147.194
--webserver_port=0
--tserver_master_addrs=127.14.147.254:44097
--keytab_file=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790722213368738-14927-0/krb5kdc/kudu_127.14.147.194.keytab
--principal=kudu/127.14.147.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.147.212:41825
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790722213368738-14927-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790722213368738-14927-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790722213368738-14927-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 22:50:31.309769 18682 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 22:50:31.309985 18682 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 22:50:31.310010 18682 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 22:50:31.311195 18682 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 22:50:31.311234 18682 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 22:50:31.311259 18682 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.147.194
I20260929 22:50:31.312841 18682 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.147.212:41825
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790722213368738-14927-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790722213368738-14927-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790722213368738-14927-0/krb5kdc/kudu_127.14.147.194.keytab
--principal=kudu/127.14.147.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.147.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790722213368738-14927-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.14.147.194
--webserver_port=0
--tserver_master_addrs=127.14.147.254:44097
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18682
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.147.194
--log_dir=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790722213368738-14927-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 21:58:54 UTC on ccycloud.quasar-xguoti.root.comops.site
build id 4681390
I20260929 22:50:31.313046 18682 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 22:50:31.313244 18682 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 22:50:31.318050 18688 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 22:50:31.318050 18690 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 22:50:31.318056 18687 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 22:50:31.318313 18682 server_base.cc:1034] running on GCE node
I20260929 22:50:31.318506 18682 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 22:50:31.318709 18682 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 22:50:31.319844 18682 hybrid_clock.cc:648] HybridClock initialized: now 1790722231319821 us; error 32 us; skew 500 ppm
Sep 29 22:50:31 dist-test-slave-98mk krb5kdc[18441](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790722231, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.147.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 22:50:31.322340 18682 init.cc:377] Logged in from keytab as kudu/127.14.147.194@KRBTEST.COM (short username kudu)
I20260929 22:50:31.322748 18682 webserver.cc:460] Webserver started at http://127.14.147.194:37123/ using document root <none> and password file <none>
I20260929 22:50:31.322965 18682 fs_manager.cc:359] Metadata directory not provided
I20260929 22:50:31.323015 18682 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 22:50:31.323137 18682 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 22:50:31.324224 18682 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790722213368738-14927-0/minicluster-data/ts-1/data/instance:
uuid: "4a1e5567d1134b6687151e88eeb90daf"
format_stamp: "Formatted at 2026-09-29 22:50:31 on dist-test-slave-98mk"
server_key: "41fdecd5269a23e7ae644b38f846ce97"
server_key_iv: "3d0f1eef8b43541e12e86625fbbf4e6f"
server_key_version: "encryptionkey@0"
I20260929 22:50:31.324548 18682 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790722213368738-14927-0/minicluster-data/ts-1/wal/instance:
uuid: "4a1e5567d1134b6687151e88eeb90daf"
format_stamp: "Formatted at 2026-09-29 22:50:31 on dist-test-slave-98mk"
server_key: "41fdecd5269a23e7ae644b38f846ce97"
server_key_iv: "3d0f1eef8b43541e12e86625fbbf4e6f"
server_key_version: "encryptionkey@0"
I20260929 22:50:31.325775 18682 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260929 22:50:31.326401 18697 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 22:50:31.326545 18682 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260929 22:50:31.326622 18682 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790722213368738-14927-0/minicluster-data/ts-1/data,/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790722213368738-14927-0/minicluster-data/ts-1/wal
uuid: "4a1e5567d1134b6687151e88eeb90daf"
format_stamp: "Formatted at 2026-09-29 22:50:31 on dist-test-slave-98mk"
server_key: "41fdecd5269a23e7ae644b38f846ce97"
server_key_iv: "3d0f1eef8b43541e12e86625fbbf4e6f"
server_key_version: "encryptionkey@0"
I20260929 22:50:31.326680 18682 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790722213368738-14927-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790722213368738-14927-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790722213368738-14927-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 22:50:31.349344 18682 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 22:50:31.351567 18682 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 22:50:31.351716 18682 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 22:50:31.351958 18682 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 22:50:31.352293 18682 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 22:50:31.352326 18682 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 22:50:31.352356 18682 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 22:50:31.352372 18682 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 22:50:31.357976 18682 rpc_server.cc:307] RPC server started. Bound to: 127.14.147.194:41431
I20260929 22:50:31.358019 18810 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.147.194:41431 every 8 connection(s)
I20260929 22:50:31.358343 18682 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790722213368738-14927-0/minicluster-data/ts-1/data/info.pb
I20260929 22:50:31.363612 14927 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhWOCLk/build/release/bin/kudu as pid 18682
I20260929 22:50:31.363726 14927 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790722213368738-14927-0/minicluster-data/ts-1/wal/instance
I20260929 22:50:31.363926 14927 external_mini_cluster.cc:1428] Setting key 6bd7c6ff0cb009cd844e6112d26ce4bd
Sep 29 22:50:31 dist-test-slave-98mk krb5kdc[18441](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790722231, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.147.194@KRBTEST.COM for kudu/127.14.147.254@KRBTEST.COM
I20260929 22:50:31.368384 18545 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:50:31.359090 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.147.194:44827 (local address 127.14.147.254:44097)
0929 22:50:31.359144 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0929 22:50:31.359147 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 22:50:31.359342 (+ 195us) server_negotiation.cc:407] Connection header received
0929 22:50:31.360336 (+ 994us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 22:50:31.360338 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 22:50:31.360348 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 22:50:31.360391 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0929 22:50:31.361015 (+ 624us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:50:31.361521 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:50:31.362642 (+ 1121us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:50:31.362905 (+ 263us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:50:31.363016 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 22:50:31.365488 (+ 2472us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 22:50:31.365489 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 22:50:31.365490 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 22:50:31.365498 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0929 22:50:31.366810 (+ 1312us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 22:50:31.367292 (+ 482us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 22:50:31.367293 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 22:50:31.367293 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 22:50:31.367360 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 22:50:31.367649 (+ 289us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 22:50:31.367650 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 22:50:31.367650 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 22:50:31.367767 (+ 117us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 22:50:31.367815 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0929 22:50:31.368177 (+ 362us) server_negotiation.cc:299] Negotiation successful
0929 22:50:31.368279 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260929 22:50:31.368935 18813 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:50:31.359065 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.147.254:44097 (local address 127.14.147.194:44827)
0929 22:50:31.359219 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0929 22:50:31.359257 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0929 22:50:31.360224 (+ 967us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 22:50:31.360465 (+ 241us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 22:50:31.360468 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 22:50:31.360743 (+ 275us) client_negotiation.cc:189] Negotiated authn=SASL
0929 22:50:31.360946 (+ 203us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 22:50:31.360951 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 22:50:31.361624 (+ 673us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 22:50:31.361625 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 22:50:31.362480 (+ 855us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 22:50:31.362483 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 22:50:31.363028 (+ 545us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 22:50:31.363029 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 22:50:31.363140 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 22:50:31.363740 (+ 600us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 22:50:31.363759 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0929 22:50:31.365386 (+ 1627us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 22:50:31.366895 (+ 1509us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 22:50:31.366903 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 22:50:31.366906 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 22:50:31.367198 (+ 292us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 22:50:31.367447 (+ 249us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 22:50:31.367447 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 22:50:31.367448 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 22:50:31.367585 (+ 137us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 22:50:31.367820 (+ 235us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 22:50:31.367825 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 22:50:31.367998 (+ 173us) client_negotiation.cc:769] Sending connection context
0929 22:50:31.368491 (+ 493us) client_negotiation.cc:240] Negotiation successful
0929 22:50:31.368686 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":78,"threads_started":1}
I20260929 22:50:31.369673 18811 heartbeater.cc:347] Connected to a master server at 127.14.147.254:44097
I20260929 22:50:31.369769 18811 heartbeater.cc:464] Registering TS with master...
I20260929 22:50:31.369946 18811 heartbeater.cc:511] Master 127.14.147.254:44097 requested a full tablet report, sending...
I20260929 22:50:31.370339 18486 ts_manager.cc:194] Registered new tserver with Master: 4a1e5567d1134b6687151e88eeb90daf (127.14.147.194:41431)
I20260929 22:50:31.370716 18486 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.147.194@KRBTEST.COM'} at 127.14.147.194:44827
WARNING: no policy specified for kudu/127.14.147.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.147.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.147.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790722213368738-14927-0/krb5kdc/kudu_127.14.147.195.keytab.
Entry for principal kudu/127.14.147.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790722213368738-14927-0/krb5kdc/kudu_127.14.147.195.keytab.
I20260929 22:50:31.388336 14927 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhWOCLk/build/release/bin/kudu
/tmp/dist-test-taskhWOCLk/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790722213368738-14927-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790722213368738-14927-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790722213368738-14927-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790722213368738-14927-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.147.195:0
--local_ip_for_outbound_sockets=127.14.147.195
--webserver_interface=127.14.147.195
--webserver_port=0
--tserver_master_addrs=127.14.147.254:44097
--keytab_file=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790722213368738-14927-0/krb5kdc/kudu_127.14.147.195.keytab
--principal=kudu/127.14.147.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.147.212:41825
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790722213368738-14927-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790722213368738-14927-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790722213368738-14927-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 22:50:31.467585 18816 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 22:50:31.467793 18816 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 22:50:31.467826 18816 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 22:50:31.469020 18816 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 22:50:31.469064 18816 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 22:50:31.469100 18816 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.147.195
I20260929 22:50:31.470654 18816 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.147.212:41825
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790722213368738-14927-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790722213368738-14927-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790722213368738-14927-0/krb5kdc/kudu_127.14.147.195.keytab
--principal=kudu/127.14.147.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.147.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790722213368738-14927-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.14.147.195
--webserver_port=0
--tserver_master_addrs=127.14.147.254:44097
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18816
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.147.195
--log_dir=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790722213368738-14927-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 21:58:54 UTC on ccycloud.quasar-xguoti.root.comops.site
build id 4681390
I20260929 22:50:31.470902 18816 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 22:50:31.471119 18816 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 22:50:31.476186 18822 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 22:50:31.476161 18821 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 22:50:31.476161 18824 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 22:50:31.476204 18816 server_base.cc:1034] running on GCE node
I20260929 22:50:31.476523 18816 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 22:50:31.476825 18816 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 22:50:31.477968 18816 hybrid_clock.cc:648] HybridClock initialized: now 1790722231477953 us; error 34 us; skew 500 ppm
Sep 29 22:50:31 dist-test-slave-98mk krb5kdc[18441](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790722231, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.147.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 22:50:31.480429 18816 init.cc:377] Logged in from keytab as kudu/127.14.147.195@KRBTEST.COM (short username kudu)
I20260929 22:50:31.480854 18816 webserver.cc:460] Webserver started at http://127.14.147.195:45805/ using document root <none> and password file <none>
I20260929 22:50:31.481050 18816 fs_manager.cc:359] Metadata directory not provided
I20260929 22:50:31.481088 18816 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 22:50:31.481176 18816 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 22:50:31.482147 18816 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790722213368738-14927-0/minicluster-data/ts-2/data/instance:
uuid: "1c875599b6ac43ca90ab33020aabcdc3"
format_stamp: "Formatted at 2026-09-29 22:50:31 on dist-test-slave-98mk"
server_key: "f8042e5b88b25bd8b4bc951dfc3b0ba2"
server_key_iv: "1a377d72fb1472e93b87acebb40ee2bb"
server_key_version: "encryptionkey@0"
I20260929 22:50:31.482432 18816 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790722213368738-14927-0/minicluster-data/ts-2/wal/instance:
uuid: "1c875599b6ac43ca90ab33020aabcdc3"
format_stamp: "Formatted at 2026-09-29 22:50:31 on dist-test-slave-98mk"
server_key: "f8042e5b88b25bd8b4bc951dfc3b0ba2"
server_key_iv: "1a377d72fb1472e93b87acebb40ee2bb"
server_key_version: "encryptionkey@0"
I20260929 22:50:31.483743 18816 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260929 22:50:31.484391 18831 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 22:50:31.484544 18816 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260929 22:50:31.484633 18816 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790722213368738-14927-0/minicluster-data/ts-2/data,/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790722213368738-14927-0/minicluster-data/ts-2/wal
uuid: "1c875599b6ac43ca90ab33020aabcdc3"
format_stamp: "Formatted at 2026-09-29 22:50:31 on dist-test-slave-98mk"
server_key: "f8042e5b88b25bd8b4bc951dfc3b0ba2"
server_key_iv: "1a377d72fb1472e93b87acebb40ee2bb"
server_key_version: "encryptionkey@0"
I20260929 22:50:31.484690 18816 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790722213368738-14927-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790722213368738-14927-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790722213368738-14927-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 22:50:31.518374 18816 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 22:50:31.520313 18816 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 22:50:31.520435 18816 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 22:50:31.520648 18816 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 22:50:31.520988 18816 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 22:50:31.521026 18816 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 22:50:31.521063 18816 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 22:50:31.521087 18816 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 22:50:31.527050 18816 rpc_server.cc:307] RPC server started. Bound to: 127.14.147.195:37395
I20260929 22:50:31.527129 18944 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.147.195:37395 every 8 connection(s)
I20260929 22:50:31.527365 18816 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790722213368738-14927-0/minicluster-data/ts-2/data/info.pb
I20260929 22:50:31.533411 14927 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhWOCLk/build/release/bin/kudu as pid 18816
I20260929 22:50:31.533576 14927 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790722213368738-14927-0/minicluster-data/ts-2/wal/instance
I20260929 22:50:31.533774 14927 external_mini_cluster.cc:1428] Setting key d22e0471a29871f29e96bf37d6112188
Sep 29 22:50:31 dist-test-slave-98mk krb5kdc[18441](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790722231, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.147.195@KRBTEST.COM for kudu/127.14.147.254@KRBTEST.COM
I20260929 22:50:31.537521 18545 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:50:31.528082 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.147.195:51451 (local address 127.14.147.254:44097)
0929 22:50:31.528159 (+ 77us) server_negotiation.cc:206] Beginning negotiation
0929 22:50:31.528162 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 22:50:31.528331 (+ 169us) server_negotiation.cc:407] Connection header received
0929 22:50:31.529236 (+ 905us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 22:50:31.529237 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 22:50:31.529247 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 22:50:31.529346 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0929 22:50:31.530015 (+ 669us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:50:31.530571 (+ 556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:50:31.531595 (+ 1024us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:50:31.531789 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:50:31.531850 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 22:50:31.534637 (+ 2787us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 22:50:31.534639 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 22:50:31.534640 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 22:50:31.534651 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0929 22:50:31.536132 (+ 1481us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 22:50:31.536589 (+ 457us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 22:50:31.536590 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 22:50:31.536590 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 22:50:31.536667 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 22:50:31.536931 (+ 264us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 22:50:31.536931 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 22:50:31.536931 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 22:50:31.537043 (+ 112us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 22:50:31.537091 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0929 22:50:31.537382 (+ 291us) server_negotiation.cc:299] Negotiation successful
0929 22:50:31.537452 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260929 22:50:31.537745 18947 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:50:31.528053 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.147.254:44097 (local address 127.14.147.195:51451)
0929 22:50:31.528209 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0929 22:50:31.528252 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0929 22:50:31.529140 (+ 888us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 22:50:31.529455 (+ 315us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 22:50:31.529459 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 22:50:31.529725 (+ 266us) client_negotiation.cc:189] Negotiated authn=SASL
0929 22:50:31.529918 (+ 193us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 22:50:31.529922 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 22:50:31.530657 (+ 735us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 22:50:31.530657 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 22:50:31.531509 (+ 852us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 22:50:31.531512 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 22:50:31.531864 (+ 352us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 22:50:31.531865 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 22:50:31.531993 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 22:50:31.532547 (+ 554us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 22:50:31.532559 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0929 22:50:31.534536 (+ 1977us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 22:50:31.536221 (+ 1685us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 22:50:31.536229 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 22:50:31.536231 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 22:50:31.536491 (+ 260us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 22:50:31.536751 (+ 260us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 22:50:31.536752 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 22:50:31.536752 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 22:50:31.536873 (+ 121us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 22:50:31.537122 (+ 249us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 22:50:31.537129 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 22:50:31.537267 (+ 138us) client_negotiation.cc:769] Sending connection context
0929 22:50:31.537366 (+ 99us) client_negotiation.cc:240] Negotiation successful
0929 22:50:31.537510 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":76,"threads_started":1}
I20260929 22:50:31.538199 18945 heartbeater.cc:347] Connected to a master server at 127.14.147.254:44097
I20260929 22:50:31.538303 18945 heartbeater.cc:464] Registering TS with master...
I20260929 22:50:31.538573 18945 heartbeater.cc:511] Master 127.14.147.254:44097 requested a full tablet report, sending...
I20260929 22:50:31.538967 18486 ts_manager.cc:194] Registered new tserver with Master: 1c875599b6ac43ca90ab33020aabcdc3 (127.14.147.195:37395)
I20260929 22:50:31.539358 18486 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.147.195@KRBTEST.COM'} at 127.14.147.195:51451
I20260929 22:50:31.546124 14927 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 29 22:50:31 dist-test-slave-98mk krb5kdc[18441](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790722231, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 29 22:50:31 dist-test-slave-98mk krb5kdc[18441](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790722231, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.147.193@KRBTEST.COM
I20260929 22:50:31.569861 18952 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:50:31.561548 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52356 (local address 127.14.147.193:36245)
0929 22:50:31.561730 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0929 22:50:31.561736 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 22:50:31.561747 (+ 11us) server_negotiation.cc:407] Connection header received
0929 22:50:31.561775 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 22:50:31.561777 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 22:50:31.561791 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 22:50:31.561858 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0929 22:50:31.562380 (+ 522us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:50:31.563103 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:50:31.563855 (+ 752us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:50:31.564189 (+ 334us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:50:31.564240 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 22:50:31.566445 (+ 2205us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 22:50:31.566455 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 22:50:31.566455 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 22:50:31.566462 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0929 22:50:31.568353 (+ 1891us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 22:50:31.568829 (+ 476us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 22:50:31.568831 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 22:50:31.568831 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 22:50:31.568895 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 22:50:31.569244 (+ 349us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 22:50:31.569245 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 22:50:31.569245 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 22:50:31.569410 (+ 165us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 22:50:31.569527 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0929 22:50:31.569697 (+ 170us) server_negotiation.cc:299] Negotiation successful
0929 22:50:31.569775 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":141,"thread_start_us":91,"threads_started":1}
Sep 29 22:50:31 dist-test-slave-98mk krb5kdc[18441](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790722231, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.147.254@KRBTEST.COM
I20260929 22:50:31.577644 18545 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:50:31.571288 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44250 (local address 127.14.147.254:44097)
0929 22:50:31.571351 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0929 22:50:31.571352 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0929 22:50:31.571361 (+ 9us) server_negotiation.cc:407] Connection header received
0929 22:50:31.571408 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 22:50:31.571409 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 22:50:31.571452 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 22:50:31.571501 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0929 22:50:31.571956 (+ 455us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:50:31.572439 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:50:31.573130 (+ 691us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:50:31.573314 (+ 184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:50:31.573365 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 22:50:31.575475 (+ 2110us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 22:50:31.575477 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 22:50:31.575477 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 22:50:31.575484 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0929 22:50:31.576593 (+ 1109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 22:50:31.576898 (+ 305us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 22:50:31.576900 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 22:50:31.576900 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 22:50:31.576971 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 22:50:31.577173 (+ 202us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 22:50:31.577174 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 22:50:31.577175 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 22:50:31.577336 (+ 161us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 22:50:31.577413 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0929 22:50:31.577500 (+ 87us) server_negotiation.cc:299] Negotiation successful
0929 22:50:31.577566 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
W20260929 22:50:31.577905 18486 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44250
I20260929 22:50:31.578233 14927 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhWOCLk/build/release/bin/kudu with pid 18548
I20260929 22:50:31.583173 14927 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhWOCLk/build/release/bin/kudu with pid 18682
I20260929 22:50:31.587474 14927 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhWOCLk/build/release/bin/kudu with pid 18816
I20260929 22:50:31.591365 14927 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhWOCLk/build/release/bin/kudu with pid 18454
2026-09-29T22:50:31Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizedSuperuser (1762 ms)
[ RUN ] SecurityITest.TestDisableWebUI
Loading random data
Initializing database '/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790722213368738-14927-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 22:50:31 dist-test-slave-98mk krb5kdc[18957](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 22:50:31 dist-test-slave-98mk krb5kdc[18957](info): set up 2 sockets
Sep 29 22:50:31 dist-test-slave-98mk krb5kdc[18957](info): commencing operation
krb5kdc: starting...
W20260929 22:50:32.547914 14927 mini_kdc.cc:121] Time spent starting KDC: real 0.936s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790722213368738-14927-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790722213368738-14927-0/krb5kdc/test-user.keytab.
Sep 29 22:50:32 dist-test-slave-98mk krb5kdc[18957](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790722232, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T22:50:32Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T22:50:32Z Disabled control of system clock
WARNING: no policy specified for kudu/127.14.147.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.147.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.147.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790722213368738-14927-0/krb5kdc/kudu_127.14.147.254.keytab.
Entry for principal kudu/127.14.147.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790722213368738-14927-0/krb5kdc/kudu_127.14.147.254.keytab.
I20260929 22:50:32.659969 14927 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhWOCLk/build/release/bin/kudu
/tmp/dist-test-taskhWOCLk/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790722213368738-14927-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790722213368738-14927-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790722213368738-14927-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790722213368738-14927-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.147.254:45519
--webserver_interface=127.14.147.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790722213368738-14927-0/krb5kdc/kudu_127.14.147.254.keytab
--principal=kudu/127.14.147.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.147.212:37699
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.147.254:45519
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790722213368738-14927-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790722213368738-14927-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790722213368738-14927-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 22:50:32.740386 18970 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 22:50:32.740612 18970 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 22:50:32.740645 18970 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 22:50:32.741863 18970 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 22:50:32.741909 18970 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 22:50:32.741927 18970 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 22:50:32.741945 18970 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 22:50:32.741958 18970 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 22:50:32.743602 18970 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.147.212:37699
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790722213368738-14927-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790722213368738-14927-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.14.147.254:45519
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790722213368738-14927-0/krb5kdc/kudu_127.14.147.254.keytab
--principal=kudu/127.14.147.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.147.254:45519
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790722213368738-14927-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.14.147.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18970
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790722213368738-14927-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 21:58:54 UTC on ccycloud.quasar-xguoti.root.comops.site
build id 4681390
I20260929 22:50:32.743865 18970 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 22:50:32.744082 18970 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 22:50:32.748773 18976 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 22:50:32.748773 18978 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 22:50:32.748940 18970 server_base.cc:1034] running on GCE node
W20260929 22:50:32.748859 18975 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 22:50:32.749208 18970 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 22:50:32.749471 18970 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 22:50:32.750643 18970 hybrid_clock.cc:648] HybridClock initialized: now 1790722232750619 us; error 37 us; skew 500 ppm
Sep 29 22:50:32 dist-test-slave-98mk krb5kdc[18957](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790722232, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.147.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 22:50:32.753311 18970 init.cc:377] Logged in from keytab as kudu/127.14.147.254@KRBTEST.COM (short username kudu)
I20260929 22:50:32.753834 18970 fs_manager.cc:359] Metadata directory not provided
I20260929 22:50:32.753937 18970 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 22:50:32.754106 18970 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 22:50:32.755764 18970 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790722213368738-14927-0/minicluster-data/master-0/data/instance:
uuid: "1f1dfe4cc07b4914b2ae1be28084c996"
format_stamp: "Formatted at 2026-09-29 22:50:32 on dist-test-slave-98mk"
server_key: "53d59c7b6250261553f10028f6297488"
server_key_iv: "f63accc673544cabfcc4dab1b3d6dc95"
server_key_version: "encryptionkey@0"
I20260929 22:50:32.756160 18970 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790722213368738-14927-0/minicluster-data/master-0/wal/instance:
uuid: "1f1dfe4cc07b4914b2ae1be28084c996"
format_stamp: "Formatted at 2026-09-29 22:50:32 on dist-test-slave-98mk"
server_key: "53d59c7b6250261553f10028f6297488"
server_key_iv: "f63accc673544cabfcc4dab1b3d6dc95"
server_key_version: "encryptionkey@0"
I20260929 22:50:32.757443 18970 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260929 22:50:32.758149 18984 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 22:50:32.758298 18970 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260929 22:50:32.758376 18970 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790722213368738-14927-0/minicluster-data/master-0/data,/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790722213368738-14927-0/minicluster-data/master-0/wal
uuid: "1f1dfe4cc07b4914b2ae1be28084c996"
format_stamp: "Formatted at 2026-09-29 22:50:32 on dist-test-slave-98mk"
server_key: "53d59c7b6250261553f10028f6297488"
server_key_iv: "f63accc673544cabfcc4dab1b3d6dc95"
server_key_version: "encryptionkey@0"
I20260929 22:50:32.758451 18970 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790722213368738-14927-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790722213368738-14927-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790722213368738-14927-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 22:50:32.779935 18970 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 22:50:32.782048 18970 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 22:50:32.782187 18970 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 22:50:32.785962 18970 rpc_server.cc:307] RPC server started. Bound to: 127.14.147.254:45519
I20260929 22:50:32.786003 19036 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.147.254:45519 every 8 connection(s)
I20260929 22:50:32.786273 18970 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790722213368738-14927-0/minicluster-data/master-0/data/info.pb
I20260929 22:50:32.787169 19037 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 22:50:32.789501 19037 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1f1dfe4cc07b4914b2ae1be28084c996: Bootstrap starting.
I20260929 22:50:32.790132 19037 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1f1dfe4cc07b4914b2ae1be28084c996: Neither blocks nor log segments found. Creating new log.
I20260929 22:50:32.790395 19037 log.cc:824] T 00000000000000000000000000000000 P 1f1dfe4cc07b4914b2ae1be28084c996: Log is configured to *not* fsync() on all Append() calls
I20260929 22:50:32.791067 19037 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1f1dfe4cc07b4914b2ae1be28084c996: No bootstrap required, opened a new log
I20260929 22:50:32.792701 19037 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1f1dfe4cc07b4914b2ae1be28084c996 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1f1dfe4cc07b4914b2ae1be28084c996" member_type: VOTER last_known_addr { host: "127.14.147.254" port: 45519 } }
I20260929 22:50:32.792855 19037 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1f1dfe4cc07b4914b2ae1be28084c996 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 22:50:32.792888 19037 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1f1dfe4cc07b4914b2ae1be28084c996 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1f1dfe4cc07b4914b2ae1be28084c996, State: Initialized, Role: FOLLOWER
I20260929 22:50:32.793022 19037 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1f1dfe4cc07b4914b2ae1be28084c996 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1f1dfe4cc07b4914b2ae1be28084c996" member_type: VOTER last_known_addr { host: "127.14.147.254" port: 45519 } }
I20260929 22:50:32.793099 19037 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1f1dfe4cc07b4914b2ae1be28084c996 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 22:50:32.793131 19037 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1f1dfe4cc07b4914b2ae1be28084c996 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 22:50:32.793180 19037 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1f1dfe4cc07b4914b2ae1be28084c996 [term 0 FOLLOWER]: Advancing to term 1
I20260929 22:50:32.794003 19037 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1f1dfe4cc07b4914b2ae1be28084c996 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1f1dfe4cc07b4914b2ae1be28084c996" member_type: VOTER last_known_addr { host: "127.14.147.254" port: 45519 } }
I20260929 22:50:32.794101 19037 leader_election.cc:302] T 00000000000000000000000000000000 P 1f1dfe4cc07b4914b2ae1be28084c996 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1f1dfe4cc07b4914b2ae1be28084c996; no voters:
I20260929 22:50:32.794272 19037 leader_election.cc:288] T 00000000000000000000000000000000 P 1f1dfe4cc07b4914b2ae1be28084c996 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 22:50:32.794335 19040 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1f1dfe4cc07b4914b2ae1be28084c996 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 22:50:32.794503 19040 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1f1dfe4cc07b4914b2ae1be28084c996 [term 1 LEADER]: Becoming Leader. State: Replica: 1f1dfe4cc07b4914b2ae1be28084c996, State: Running, Role: LEADER
I20260929 22:50:32.794514 19037 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1f1dfe4cc07b4914b2ae1be28084c996 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 22:50:32.794610 19040 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1f1dfe4cc07b4914b2ae1be28084c996 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1f1dfe4cc07b4914b2ae1be28084c996" member_type: VOTER last_known_addr { host: "127.14.147.254" port: 45519 } }
I20260929 22:50:32.795097 14927 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhWOCLk/build/release/bin/kudu as pid 18970
I20260929 22:50:32.795213 14927 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790722213368738-14927-0/minicluster-data/master-0/wal/instance
I20260929 22:50:32.795217 19041 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1f1dfe4cc07b4914b2ae1be28084c996 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1f1dfe4cc07b4914b2ae1be28084c996" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1f1dfe4cc07b4914b2ae1be28084c996" member_type: VOTER last_known_addr { host: "127.14.147.254" port: 45519 } } }
I20260929 22:50:32.795244 19042 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1f1dfe4cc07b4914b2ae1be28084c996 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1f1dfe4cc07b4914b2ae1be28084c996. Latest consensus state: current_term: 1 leader_uuid: "1f1dfe4cc07b4914b2ae1be28084c996" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1f1dfe4cc07b4914b2ae1be28084c996" member_type: VOTER last_known_addr { host: "127.14.147.254" port: 45519 } } }
I20260929 22:50:32.795298 19041 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1f1dfe4cc07b4914b2ae1be28084c996 [sys.catalog]: This master's current role is: LEADER
I20260929 22:50:32.795301 19042 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1f1dfe4cc07b4914b2ae1be28084c996 [sys.catalog]: This master's current role is: LEADER
I20260929 22:50:32.795432 14927 external_mini_cluster.cc:1428] Setting key 79ffb651487a0c3f79db2a02dc035ea2
I20260929 22:50:32.795671 19049 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 22:50:32.796190 19049 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 22:50:32.797713 19049 catalog_manager.cc:1347] Generated new cluster ID: 2f2b644d147c44b79cb3b53f16f00c70
I20260929 22:50:32.797766 19049 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 29 22:50:32 dist-test-slave-98mk krb5kdc[18957](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790722232, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.147.254@KRBTEST.COM
I20260929 22:50:32.805781 19059 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:50:32.796054 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59494 (local address 127.14.147.254:45519)
0929 22:50:32.796186 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0929 22:50:32.796196 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 22:50:32.796226 (+ 30us) server_negotiation.cc:407] Connection header received
0929 22:50:32.796351 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 22:50:32.796359 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 22:50:32.796405 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 22:50:32.796531 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0929 22:50:32.797181 (+ 650us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:50:32.799392 (+ 2211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:50:32.800131 (+ 739us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:50:32.800575 (+ 444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:50:32.800670 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 22:50:32.802791 (+ 2121us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 22:50:32.802796 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 22:50:32.802796 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 22:50:32.802809 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0929 22:50:32.804331 (+ 1522us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 22:50:32.804651 (+ 320us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 22:50:32.804653 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 22:50:32.804654 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 22:50:32.804743 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 22:50:32.804923 (+ 180us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 22:50:32.804924 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 22:50:32.804924 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 22:50:32.805161 (+ 237us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 22:50:32.805315 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0929 22:50:32.805385 (+ 70us) server_negotiation.cc:299] Negotiation successful
0929 22:50:32.805612 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86,"thread_start_us":66,"threads_started":1}
I20260929 22:50:32.833293 19049 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 22:50:32.833942 19049 catalog_manager.cc:1504] Loading token signing keys...
I20260929 22:50:32.839818 19049 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1f1dfe4cc07b4914b2ae1be28084c996: Generated new TSK 0
I20260929 22:50:32.840015 19049 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.14.147.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.147.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.147.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790722213368738-14927-0/krb5kdc/kudu_127.14.147.193.keytab.
Entry for principal kudu/127.14.147.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790722213368738-14927-0/krb5kdc/kudu_127.14.147.193.keytab.
I20260929 22:50:32.867600 14927 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhWOCLk/build/release/bin/kudu
/tmp/dist-test-taskhWOCLk/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790722213368738-14927-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790722213368738-14927-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790722213368738-14927-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790722213368738-14927-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.147.193:0
--local_ip_for_outbound_sockets=127.14.147.193
--webserver_interface=127.14.147.193
--webserver_port=0
--tserver_master_addrs=127.14.147.254:45519
--keytab_file=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790722213368738-14927-0/krb5kdc/kudu_127.14.147.193.keytab
--principal=kudu/127.14.147.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.147.212:37699
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790722213368738-14927-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790722213368738-14927-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790722213368738-14927-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 22:50:32.949808 19063 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 22:50:32.950033 19063 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 22:50:32.950068 19063 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 22:50:32.951349 19063 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 22:50:32.951396 19063 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 22:50:32.951465 19063 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.147.193
I20260929 22:50:32.953181 19063 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.147.212:37699
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790722213368738-14927-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790722213368738-14927-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790722213368738-14927-0/krb5kdc/kudu_127.14.147.193.keytab
--principal=kudu/127.14.147.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.147.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790722213368738-14927-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.14.147.193
--webserver_port=0
--tserver_master_addrs=127.14.147.254:45519
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19063
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.147.193
--log_dir=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790722213368738-14927-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 21:58:54 UTC on ccycloud.quasar-xguoti.root.comops.site
build id 4681390
I20260929 22:50:32.953437 19063 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 22:50:32.953660 19063 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 22:50:32.958595 19068 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 22:50:32.958595 19069 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 22:50:32.958644 19063 server_base.cc:1034] running on GCE node
W20260929 22:50:32.958595 19071 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 22:50:32.959031 19063 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 22:50:32.959278 19063 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 22:50:32.960433 19063 hybrid_clock.cc:648] HybridClock initialized: now 1790722232960418 us; error 31 us; skew 500 ppm
Sep 29 22:50:32 dist-test-slave-98mk krb5kdc[18957](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790722232, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.147.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 22:50:32.963110 19063 init.cc:377] Logged in from keytab as kudu/127.14.147.193@KRBTEST.COM (short username kudu)
I20260929 22:50:32.963683 19063 fs_manager.cc:359] Metadata directory not provided
I20260929 22:50:32.963764 19063 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 22:50:32.963901 19063 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 22:50:32.964999 19063 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790722213368738-14927-0/minicluster-data/ts-0/data/instance:
uuid: "7beb8031dfff47bd97c0ff3257a0f433"
format_stamp: "Formatted at 2026-09-29 22:50:32 on dist-test-slave-98mk"
server_key: "0e9277d7ef703c15b10d59b8f90d58d2"
server_key_iv: "d98d2a5dbe861a5acbc3af02a5d2dd34"
server_key_version: "encryptionkey@0"
I20260929 22:50:32.965353 19063 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790722213368738-14927-0/minicluster-data/ts-0/wal/instance:
uuid: "7beb8031dfff47bd97c0ff3257a0f433"
format_stamp: "Formatted at 2026-09-29 22:50:32 on dist-test-slave-98mk"
server_key: "0e9277d7ef703c15b10d59b8f90d58d2"
server_key_iv: "d98d2a5dbe861a5acbc3af02a5d2dd34"
server_key_version: "encryptionkey@0"
I20260929 22:50:32.966804 19063 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260929 22:50:32.967885 19077 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 22:50:32.968116 19063 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260929 22:50:32.968223 19063 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790722213368738-14927-0/minicluster-data/ts-0/data,/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790722213368738-14927-0/minicluster-data/ts-0/wal
uuid: "7beb8031dfff47bd97c0ff3257a0f433"
format_stamp: "Formatted at 2026-09-29 22:50:32 on dist-test-slave-98mk"
server_key: "0e9277d7ef703c15b10d59b8f90d58d2"
server_key_iv: "d98d2a5dbe861a5acbc3af02a5d2dd34"
server_key_version: "encryptionkey@0"
I20260929 22:50:32.968297 19063 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790722213368738-14927-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790722213368738-14927-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790722213368738-14927-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 22:50:32.990762 19063 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 22:50:32.993301 19063 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 22:50:32.993525 19063 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 22:50:32.993824 19063 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 22:50:32.994196 19063 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 22:50:32.994231 19063 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 22:50:32.994264 19063 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 22:50:32.994285 19063 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 22:50:33.001433 19063 rpc_server.cc:307] RPC server started. Bound to: 127.14.147.193:43511
I20260929 22:50:33.001477 19190 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.147.193:43511 every 8 connection(s)
I20260929 22:50:33.001755 19063 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790722213368738-14927-0/minicluster-data/ts-0/data/info.pb
I20260929 22:50:33.002964 14927 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhWOCLk/build/release/bin/kudu as pid 19063
I20260929 22:50:33.003058 14927 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790722213368738-14927-0/minicluster-data/ts-0/wal/instance
I20260929 22:50:33.003239 14927 external_mini_cluster.cc:1428] Setting key 24b85dfdc55a163f9b277392d32772f8
Sep 29 22:50:33 dist-test-slave-98mk krb5kdc[18957](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790722232, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.147.193@KRBTEST.COM for kudu/127.14.147.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.14.147.194@KRBTEST.COM; defaulting to no policy
I20260929 22:50:33.012271 19059 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:50:33.002444 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.147.193:49857 (local address 127.14.147.254:45519)
0929 22:50:33.002489 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0929 22:50:33.002491 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 22:50:33.002697 (+ 206us) server_negotiation.cc:407] Connection header received
0929 22:50:33.003268 (+ 571us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 22:50:33.003269 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 22:50:33.003282 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 22:50:33.003332 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0929 22:50:33.004664 (+ 1332us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:50:33.005245 (+ 581us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:50:33.006228 (+ 983us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:50:33.006522 (+ 294us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:50:33.006598 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 22:50:33.009134 (+ 2536us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 22:50:33.009136 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 22:50:33.009136 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 22:50:33.009148 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0929 22:50:33.010884 (+ 1736us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 22:50:33.011282 (+ 398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 22:50:33.011284 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 22:50:33.011284 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 22:50:33.011345 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 22:50:33.011617 (+ 272us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 22:50:33.011618 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 22:50:33.011619 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 22:50:33.011773 (+ 154us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 22:50:33.011842 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0929 22:50:33.012109 (+ 267us) server_negotiation.cc:299] Negotiation successful
0929 22:50:33.012198 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260929 22:50:33.012380 19193 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:50:33.002417 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.147.254:45519 (local address 127.14.147.193:49857)
0929 22:50:33.002574 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0929 22:50:33.002611 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0929 22:50:33.003161 (+ 550us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 22:50:33.003453 (+ 292us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 22:50:33.003458 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 22:50:33.003843 (+ 385us) client_negotiation.cc:189] Negotiated authn=SASL
0929 22:50:33.004545 (+ 702us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 22:50:33.004553 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 22:50:33.005350 (+ 797us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 22:50:33.005351 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 22:50:33.006159 (+ 808us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 22:50:33.006162 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 22:50:33.006631 (+ 469us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 22:50:33.006632 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 22:50:33.006753 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 22:50:33.007315 (+ 562us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 22:50:33.007327 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0929 22:50:33.009026 (+ 1699us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 22:50:33.010970 (+ 1944us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 22:50:33.010979 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 22:50:33.010981 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 22:50:33.011208 (+ 227us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 22:50:33.011435 (+ 227us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 22:50:33.011436 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 22:50:33.011436 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 22:50:33.011566 (+ 130us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 22:50:33.011835 (+ 269us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 22:50:33.011838 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 22:50:33.011966 (+ 128us) client_negotiation.cc:769] Sending connection context
0929 22:50:33.012044 (+ 78us) client_negotiation.cc:240] Negotiation successful
0929 22:50:33.012178 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":75,"threads_started":1}
I20260929 22:50:33.012934 19191 heartbeater.cc:347] Connected to a master server at 127.14.147.254:45519
I20260929 22:50:33.013036 19191 heartbeater.cc:464] Registering TS with master...
I20260929 22:50:33.013186 19191 heartbeater.cc:511] Master 127.14.147.254:45519 requested a full tablet report, sending...
I20260929 22:50:33.013757 19001 ts_manager.cc:194] Registered new tserver with Master: 7beb8031dfff47bd97c0ff3257a0f433 (127.14.147.193:43511)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.147.194@KRBTEST.COM" created.
I20260929 22:50:33.014421 19001 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.147.193@KRBTEST.COM'} at 127.14.147.193:49857
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.147.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790722213368738-14927-0/krb5kdc/kudu_127.14.147.194.keytab.
Entry for principal kudu/127.14.147.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790722213368738-14927-0/krb5kdc/kudu_127.14.147.194.keytab.
I20260929 22:50:33.028254 14927 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhWOCLk/build/release/bin/kudu
/tmp/dist-test-taskhWOCLk/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790722213368738-14927-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790722213368738-14927-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790722213368738-14927-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790722213368738-14927-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.147.194:0
--local_ip_for_outbound_sockets=127.14.147.194
--webserver_interface=127.14.147.194
--webserver_port=0
--tserver_master_addrs=127.14.147.254:45519
--keytab_file=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790722213368738-14927-0/krb5kdc/kudu_127.14.147.194.keytab
--principal=kudu/127.14.147.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.147.212:37699
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790722213368738-14927-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790722213368738-14927-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790722213368738-14927-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 22:50:33.109261 19196 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 22:50:33.109494 19196 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 22:50:33.109520 19196 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 22:50:33.110785 19196 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 22:50:33.110839 19196 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 22:50:33.110867 19196 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.147.194
I20260929 22:50:33.112520 19196 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.147.212:37699
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790722213368738-14927-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790722213368738-14927-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790722213368738-14927-0/krb5kdc/kudu_127.14.147.194.keytab
--principal=kudu/127.14.147.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.147.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790722213368738-14927-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.14.147.194
--webserver_port=0
--tserver_master_addrs=127.14.147.254:45519
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19196
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.147.194
--log_dir=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790722213368738-14927-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 21:58:54 UTC on ccycloud.quasar-xguoti.root.comops.site
build id 4681390
I20260929 22:50:33.112777 19196 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 22:50:33.112999 19196 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 22:50:33.117918 19202 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 22:50:33.118027 19196 server_base.cc:1034] running on GCE node
W20260929 22:50:33.117909 19204 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 22:50:33.117913 19201 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 22:50:33.118364 19196 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 22:50:33.118625 19196 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 22:50:33.119762 19196 hybrid_clock.cc:648] HybridClock initialized: now 1790722233119749 us; error 30 us; skew 500 ppm
Sep 29 22:50:33 dist-test-slave-98mk krb5kdc[18957](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790722233, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.147.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 22:50:33.122285 19196 init.cc:377] Logged in from keytab as kudu/127.14.147.194@KRBTEST.COM (short username kudu)
I20260929 22:50:33.122637 19196 fs_manager.cc:359] Metadata directory not provided
I20260929 22:50:33.122682 19196 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 22:50:33.122809 19196 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 22:50:33.123986 19196 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790722213368738-14927-0/minicluster-data/ts-1/data/instance:
uuid: "5938f25392c8486fa687b3fd56847b9a"
format_stamp: "Formatted at 2026-09-29 22:50:33 on dist-test-slave-98mk"
server_key: "5e6c1430aa024604223729ed6c09a52a"
server_key_iv: "cb917357605c10ac7b42ce65e2bbba36"
server_key_version: "encryptionkey@0"
I20260929 22:50:33.124320 19196 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790722213368738-14927-0/minicluster-data/ts-1/wal/instance:
uuid: "5938f25392c8486fa687b3fd56847b9a"
format_stamp: "Formatted at 2026-09-29 22:50:33 on dist-test-slave-98mk"
server_key: "5e6c1430aa024604223729ed6c09a52a"
server_key_iv: "cb917357605c10ac7b42ce65e2bbba36"
server_key_version: "encryptionkey@0"
I20260929 22:50:33.125643 19196 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260929 22:50:33.126577 19210 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 22:50:33.126909 19196 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260929 22:50:33.127038 19196 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790722213368738-14927-0/minicluster-data/ts-1/data,/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790722213368738-14927-0/minicluster-data/ts-1/wal
uuid: "5938f25392c8486fa687b3fd56847b9a"
format_stamp: "Formatted at 2026-09-29 22:50:33 on dist-test-slave-98mk"
server_key: "5e6c1430aa024604223729ed6c09a52a"
server_key_iv: "cb917357605c10ac7b42ce65e2bbba36"
server_key_version: "encryptionkey@0"
I20260929 22:50:33.127143 19196 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790722213368738-14927-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790722213368738-14927-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790722213368738-14927-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 22:50:33.137437 19196 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 22:50:33.139549 19196 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 22:50:33.139664 19196 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 22:50:33.139866 19196 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 22:50:33.140136 19196 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 22:50:33.140164 19196 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 22:50:33.140192 19196 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 22:50:33.140206 19196 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 22:50:33.146274 19196 rpc_server.cc:307] RPC server started. Bound to: 127.14.147.194:38801
I20260929 22:50:33.146376 19323 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.147.194:38801 every 8 connection(s)
I20260929 22:50:33.146787 19196 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790722213368738-14927-0/minicluster-data/ts-1/data/info.pb
Sep 29 22:50:33 dist-test-slave-98mk krb5kdc[18957](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790722233, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.147.194@KRBTEST.COM for kudu/127.14.147.254@KRBTEST.COM
I20260929 22:50:33.153493 14927 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhWOCLk/build/release/bin/kudu as pid 19196
I20260929 22:50:33.153586 14927 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790722213368738-14927-0/minicluster-data/ts-1/wal/instance
I20260929 22:50:33.153733 14927 external_mini_cluster.cc:1428] Setting key 74463e1a80286c2e081d03c746238f00
I20260929 22:50:33.156243 19059 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:50:33.147621 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.147.194:56787 (local address 127.14.147.254:45519)
0929 22:50:33.147684 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0929 22:50:33.147686 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 22:50:33.147900 (+ 214us) server_negotiation.cc:407] Connection header received
0929 22:50:33.148397 (+ 497us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 22:50:33.148398 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 22:50:33.148409 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 22:50:33.148467 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0929 22:50:33.149398 (+ 931us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:50:33.149924 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:50:33.150974 (+ 1050us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:50:33.151183 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:50:33.151257 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 22:50:33.153342 (+ 2085us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 22:50:33.153344 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 22:50:33.153344 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 22:50:33.153352 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0929 22:50:33.154823 (+ 1471us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 22:50:33.155218 (+ 395us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 22:50:33.155219 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 22:50:33.155220 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 22:50:33.155277 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 22:50:33.155529 (+ 252us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 22:50:33.155530 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 22:50:33.155530 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 22:50:33.155662 (+ 132us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 22:50:33.155733 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0929 22:50:33.156057 (+ 324us) server_negotiation.cc:299] Negotiation successful
0929 22:50:33.156148 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260929 22:50:33.156360 19326 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:50:33.147600 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.147.254:45519 (local address 127.14.147.194:56787)
0929 22:50:33.147777 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0929 22:50:33.147817 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0929 22:50:33.148283 (+ 466us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 22:50:33.148523 (+ 240us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 22:50:33.148526 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 22:50:33.148787 (+ 261us) client_negotiation.cc:189] Negotiated authn=SASL
0929 22:50:33.149288 (+ 501us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 22:50:33.149303 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 22:50:33.150019 (+ 716us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 22:50:33.150021 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 22:50:33.150881 (+ 860us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 22:50:33.150884 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 22:50:33.151253 (+ 369us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 22:50:33.151253 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 22:50:33.151360 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 22:50:33.151868 (+ 508us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 22:50:33.151876 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0929 22:50:33.153238 (+ 1362us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 22:50:33.154896 (+ 1658us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 22:50:33.154903 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 22:50:33.154905 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 22:50:33.155125 (+ 220us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 22:50:33.155331 (+ 206us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 22:50:33.155332 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 22:50:33.155332 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 22:50:33.155474 (+ 142us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 22:50:33.155745 (+ 271us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 22:50:33.155754 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 22:50:33.155934 (+ 180us) client_negotiation.cc:769] Sending connection context
0929 22:50:33.156026 (+ 92us) client_negotiation.cc:240] Negotiation successful
0929 22:50:33.156158 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":136,"thread_start_us":105,"threads_started":1}
I20260929 22:50:33.156800 19324 heartbeater.cc:347] Connected to a master server at 127.14.147.254:45519
I20260929 22:50:33.156901 19324 heartbeater.cc:464] Registering TS with master...
I20260929 22:50:33.157054 19324 heartbeater.cc:511] Master 127.14.147.254:45519 requested a full tablet report, sending...
I20260929 22:50:33.157426 19001 ts_manager.cc:194] Registered new tserver with Master: 5938f25392c8486fa687b3fd56847b9a (127.14.147.194:38801)
I20260929 22:50:33.157801 19001 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.147.194@KRBTEST.COM'} at 127.14.147.194:56787
WARNING: no policy specified for kudu/127.14.147.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.147.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.147.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790722213368738-14927-0/krb5kdc/kudu_127.14.147.195.keytab.
Entry for principal kudu/127.14.147.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790722213368738-14927-0/krb5kdc/kudu_127.14.147.195.keytab.
I20260929 22:50:33.177340 14927 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhWOCLk/build/release/bin/kudu
/tmp/dist-test-taskhWOCLk/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790722213368738-14927-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790722213368738-14927-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790722213368738-14927-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790722213368738-14927-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.147.195:0
--local_ip_for_outbound_sockets=127.14.147.195
--webserver_interface=127.14.147.195
--webserver_port=0
--tserver_master_addrs=127.14.147.254:45519
--keytab_file=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790722213368738-14927-0/krb5kdc/kudu_127.14.147.195.keytab
--principal=kudu/127.14.147.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.147.212:37699
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790722213368738-14927-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790722213368738-14927-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790722213368738-14927-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 22:50:33.256855 19329 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 22:50:33.257078 19329 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 22:50:33.257113 19329 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 22:50:33.258370 19329 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 22:50:33.258427 19329 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 22:50:33.258466 19329 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.147.195
I20260929 22:50:33.260269 19329 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.147.212:37699
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790722213368738-14927-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790722213368738-14927-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790722213368738-14927-0/krb5kdc/kudu_127.14.147.195.keytab
--principal=kudu/127.14.147.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.147.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790722213368738-14927-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.14.147.195
--webserver_port=0
--tserver_master_addrs=127.14.147.254:45519
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19329
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.147.195
--log_dir=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790722213368738-14927-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 21:58:54 UTC on ccycloud.quasar-xguoti.root.comops.site
build id 4681390
I20260929 22:50:33.260546 19329 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 22:50:33.260788 19329 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 22:50:33.265740 19335 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 22:50:33.265740 19337 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 22:50:33.265821 19329 server_base.cc:1034] running on GCE node
W20260929 22:50:33.265789 19334 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 22:50:33.266161 19329 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 22:50:33.266417 19329 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 22:50:33.267565 19329 hybrid_clock.cc:648] HybridClock initialized: now 1790722233267552 us; error 27 us; skew 500 ppm
Sep 29 22:50:33 dist-test-slave-98mk krb5kdc[18957](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790722233, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.147.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 22:50:33.270045 19329 init.cc:377] Logged in from keytab as kudu/127.14.147.195@KRBTEST.COM (short username kudu)
I20260929 22:50:33.270433 19329 fs_manager.cc:359] Metadata directory not provided
I20260929 22:50:33.270711 19329 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 22:50:33.270855 19329 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 22:50:33.271978 19329 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790722213368738-14927-0/minicluster-data/ts-2/data/instance:
uuid: "86fcdef103bd41abba87cd27d5ebce1a"
format_stamp: "Formatted at 2026-09-29 22:50:33 on dist-test-slave-98mk"
server_key: "b21ee4ad4360db6d705055647443c49c"
server_key_iv: "db031aea6edecf754459f60dac8738c4"
server_key_version: "encryptionkey@0"
I20260929 22:50:33.272280 19329 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790722213368738-14927-0/minicluster-data/ts-2/wal/instance:
uuid: "86fcdef103bd41abba87cd27d5ebce1a"
format_stamp: "Formatted at 2026-09-29 22:50:33 on dist-test-slave-98mk"
server_key: "b21ee4ad4360db6d705055647443c49c"
server_key_iv: "db031aea6edecf754459f60dac8738c4"
server_key_version: "encryptionkey@0"
I20260929 22:50:33.273646 19329 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260929 22:50:33.274659 19343 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 22:50:33.274857 19329 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260929 22:50:33.274952 19329 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790722213368738-14927-0/minicluster-data/ts-2/data,/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790722213368738-14927-0/minicluster-data/ts-2/wal
uuid: "86fcdef103bd41abba87cd27d5ebce1a"
format_stamp: "Formatted at 2026-09-29 22:50:33 on dist-test-slave-98mk"
server_key: "b21ee4ad4360db6d705055647443c49c"
server_key_iv: "db031aea6edecf754459f60dac8738c4"
server_key_version: "encryptionkey@0"
I20260929 22:50:33.275048 19329 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790722213368738-14927-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790722213368738-14927-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790722213368738-14927-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 22:50:33.284062 19329 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 22:50:33.286132 19329 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 22:50:33.286257 19329 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 22:50:33.286484 19329 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 22:50:33.286782 19329 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 22:50:33.286818 19329 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 22:50:33.286842 19329 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 22:50:33.286865 19329 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 22:50:33.292490 19329 rpc_server.cc:307] RPC server started. Bound to: 127.14.147.195:33997
I20260929 22:50:33.292529 19456 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.147.195:33997 every 8 connection(s)
I20260929 22:50:33.292783 19329 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790722213368738-14927-0/minicluster-data/ts-2/data/info.pb
Sep 29 22:50:33 dist-test-slave-98mk krb5kdc[18957](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790722233, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.147.195@KRBTEST.COM for kudu/127.14.147.254@KRBTEST.COM
I20260929 22:50:33.302132 14927 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhWOCLk/build/release/bin/kudu as pid 19329
I20260929 22:50:33.302242 14927 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790722213368738-14927-0/minicluster-data/ts-2/wal/instance
I20260929 22:50:33.302460 14927 external_mini_cluster.cc:1428] Setting key 9834ce87694af1475a7a7f4e5e69eeb6
I20260929 22:50:33.303252 19059 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:50:33.293465 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.147.195:34569 (local address 127.14.147.254:45519)
0929 22:50:33.293510 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0929 22:50:33.293512 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 22:50:33.293733 (+ 221us) server_negotiation.cc:407] Connection header received
0929 22:50:33.294550 (+ 817us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 22:50:33.294552 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 22:50:33.294572 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 22:50:33.294634 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0929 22:50:33.296047 (+ 1413us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:50:33.296626 (+ 579us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:50:33.297719 (+ 1093us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:50:33.297918 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:50:33.297976 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 22:50:33.300485 (+ 2509us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 22:50:33.300487 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 22:50:33.300487 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 22:50:33.300497 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0929 22:50:33.301604 (+ 1107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 22:50:33.302018 (+ 414us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 22:50:33.302020 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 22:50:33.302020 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 22:50:33.302073 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 22:50:33.302356 (+ 283us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 22:50:33.302357 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 22:50:33.302358 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 22:50:33.302537 (+ 179us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 22:50:33.302632 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0929 22:50:33.303019 (+ 387us) server_negotiation.cc:299] Negotiation successful
0929 22:50:33.303143 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260929 22:50:33.303390 19459 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:50:33.293449 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.147.254:45519 (local address 127.14.147.195:34569)
0929 22:50:33.293616 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0929 22:50:33.293651 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0929 22:50:33.294376 (+ 725us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 22:50:33.294747 (+ 371us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 22:50:33.294753 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 22:50:33.295179 (+ 426us) client_negotiation.cc:189] Negotiated authn=SASL
0929 22:50:33.295884 (+ 705us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 22:50:33.295892 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 22:50:33.296720 (+ 828us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 22:50:33.296721 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 22:50:33.297632 (+ 911us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 22:50:33.297635 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 22:50:33.297992 (+ 357us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 22:50:33.297993 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 22:50:33.298140 (+ 147us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 22:50:33.298811 (+ 671us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 22:50:33.298824 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0929 22:50:33.300348 (+ 1524us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 22:50:33.301694 (+ 1346us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 22:50:33.301702 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 22:50:33.301705 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 22:50:33.301931 (+ 226us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 22:50:33.302128 (+ 197us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 22:50:33.302129 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 22:50:33.302130 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 22:50:33.302246 (+ 116us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 22:50:33.302683 (+ 437us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 22:50:33.302691 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 22:50:33.302848 (+ 157us) client_negotiation.cc:769] Sending connection context
0929 22:50:33.302966 (+ 118us) client_negotiation.cc:240] Negotiation successful
0929 22:50:33.303146 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":131,"thread_start_us":89,"threads_started":1}
I20260929 22:50:33.303941 19457 heartbeater.cc:347] Connected to a master server at 127.14.147.254:45519
I20260929 22:50:33.304025 19457 heartbeater.cc:464] Registering TS with master...
I20260929 22:50:33.304149 19457 heartbeater.cc:511] Master 127.14.147.254:45519 requested a full tablet report, sending...
I20260929 22:50:33.304466 19001 ts_manager.cc:194] Registered new tserver with Master: 86fcdef103bd41abba87cd27d5ebce1a (127.14.147.195:33997)
I20260929 22:50:33.304818 19001 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.147.195@KRBTEST.COM'} at 127.14.147.195:34569
I20260929 22:50:33.314684 14927 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 22:50:33.321365 19059 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:50:33.315942 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59502 (local address 127.14.147.254:45519)
0929 22:50:33.316000 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0929 22:50:33.316002 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 22:50:33.316108 (+ 106us) server_negotiation.cc:407] Connection header received
0929 22:50:33.316152 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 22:50:33.316153 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 22:50:33.316163 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 22:50:33.316208 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0929 22:50:33.316690 (+ 482us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:50:33.317197 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:50:33.317888 (+ 691us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:50:33.318087 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:50:33.318147 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 22:50:33.319236 (+ 1089us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 22:50:33.319237 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 22:50:33.319237 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 22:50:33.319244 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0929 22:50:33.320467 (+ 1223us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 22:50:33.320758 (+ 291us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 22:50:33.320758 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 22:50:33.320758 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 22:50:33.320803 (+ 45us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 22:50:33.320927 (+ 124us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 22:50:33.320928 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 22:50:33.320928 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 22:50:33.321069 (+ 141us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 22:50:33.321142 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0929 22:50:33.321218 (+ 76us) server_negotiation.cc:299] Negotiation successful
0929 22:50:33.321278 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260929 22:50:33.322841 19001 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59502:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 22:50:33.324496 19001 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 22:50:33.330976 19465 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:50:33.326547 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.147.194:38801 (local address 127.0.0.1:35872)
0929 22:50:33.326972 (+ 425us) negotiation.cc:107] Waiting for socket to connect
0929 22:50:33.326981 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0929 22:50:33.327020 (+ 39us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 22:50:33.327327 (+ 307us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 22:50:33.327328 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 22:50:33.327330 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 22:50:33.327359 (+ 29us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 22:50:33.327360 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 22:50:33.328356 (+ 996us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 22:50:33.328357 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 22:50:33.329805 (+ 1448us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 22:50:33.329808 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 22:50:33.330711 (+ 903us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 22:50:33.330712 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 22:50:33.330779 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 22:50:33.330784 (+ 5us) client_negotiation.cc:769] Sending connection context
0929 22:50:33.330833 (+ 49us) client_negotiation.cc:240] Negotiation successful
0929 22:50:33.330891 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":415,"mutex_wait_us":5,"thread_start_us":64,"threads_started":1}
I20260929 22:50:33.330978 19470 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:50:33.326933 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35872 (local address 127.14.147.194:38801)
0929 22:50:33.327098 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0929 22:50:33.327102 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 22:50:33.327111 (+ 9us) server_negotiation.cc:407] Connection header received
0929 22:50:33.327147 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 22:50:33.327148 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 22:50:33.327162 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 22:50:33.327454 (+ 292us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 22:50:33.327536 (+ 82us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:50:33.328251 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:50:33.330049 (+ 1798us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:50:33.330633 (+ 584us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:50:33.330711 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 22:50:33.330765 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0929 22:50:33.330855 (+ 90us) server_negotiation.cc:299] Negotiation successful
0929 22:50:33.330891 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":137,"thread_start_us":69,"threads_started":1}
I20260929 22:50:33.331063 19469 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:50:33.326813 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53582 (local address 127.14.147.195:33997)
0929 22:50:33.327114 (+ 301us) server_negotiation.cc:206] Beginning negotiation
0929 22:50:33.327119 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 22:50:33.327128 (+ 9us) server_negotiation.cc:407] Connection header received
0929 22:50:33.327165 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 22:50:33.327167 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 22:50:33.327182 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 22:50:33.327258 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 22:50:33.327463 (+ 205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:50:33.328618 (+ 1155us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:50:33.329833 (+ 1215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:50:33.330464 (+ 631us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:50:33.330535 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 22:50:33.330590 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0929 22:50:33.330938 (+ 348us) server_negotiation.cc:299] Negotiation successful
0929 22:50:33.330986 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":271,"thread_start_us":92,"threads_started":1}
I20260929 22:50:33.331220 19467 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:50:33.326542 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.147.195:33997 (local address 127.0.0.1:53582)
0929 22:50:33.326715 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0929 22:50:33.326754 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0929 22:50:33.326820 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 22:50:33.327257 (+ 437us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 22:50:33.327259 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 22:50:33.327261 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 22:50:33.327318 (+ 57us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 22:50:33.327320 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 22:50:33.328744 (+ 1424us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 22:50:33.328745 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 22:50:33.329754 (+ 1009us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 22:50:33.329756 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 22:50:33.330626 (+ 870us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 22:50:33.330626 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 22:50:33.330714 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 22:50:33.330728 (+ 14us) client_negotiation.cc:769] Sending connection context
0929 22:50:33.330793 (+ 65us) client_negotiation.cc:240] Negotiation successful
0929 22:50:33.330861 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":148,"mutex_wait_us":32,"thread_start_us":78,"threads_started":1}
I20260929 22:50:33.331378 19466 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:50:33.326547 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.147.193:43511 (local address 127.0.0.1:36884)
0929 22:50:33.326909 (+ 362us) negotiation.cc:107] Waiting for socket to connect
0929 22:50:33.326917 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0929 22:50:33.326963 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 22:50:33.327161 (+ 198us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 22:50:33.327164 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 22:50:33.327166 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 22:50:33.327261 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 22:50:33.327263 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 22:50:33.328800 (+ 1537us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 22:50:33.328802 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 22:50:33.330061 (+ 1259us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 22:50:33.330065 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 22:50:33.331096 (+ 1031us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 22:50:33.331097 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 22:50:33.331220 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 22:50:33.331225 (+ 5us) client_negotiation.cc:769] Sending connection context
0929 22:50:33.331274 (+ 49us) client_negotiation.cc:240] Negotiation successful
0929 22:50:33.331323 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":351,"mutex_wait_us":4,"thread_start_us":56,"threads_started":1}
I20260929 22:50:33.331661 19468 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:50:33.326744 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36884 (local address 127.14.147.193:43511)
0929 22:50:33.326906 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0929 22:50:33.326911 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 22:50:33.326958 (+ 47us) server_negotiation.cc:407] Connection header received
0929 22:50:33.327030 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 22:50:33.327032 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 22:50:33.327049 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 22:50:33.327141 (+ 92us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 22:50:33.327396 (+ 255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:50:33.328625 (+ 1229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:50:33.330209 (+ 1584us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:50:33.330958 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:50:33.331017 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 22:50:33.331062 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0929 22:50:33.331536 (+ 474us) server_negotiation.cc:299] Negotiation successful
0929 22:50:33.331581 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":127,"thread_start_us":79,"threads_started":1}
I20260929 22:50:33.332056 19391 tablet_service.cc:1467] Processing CreateTablet for tablet 4575e4961c2048dab05ce7613792355f (DEFAULT_TABLE table=test-table [id=5a487bbb6bb94c0f9d44ef41d922f3b3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 22:50:33.332305 19125 tablet_service.cc:1467] Processing CreateTablet for tablet 4575e4961c2048dab05ce7613792355f (DEFAULT_TABLE table=test-table [id=5a487bbb6bb94c0f9d44ef41d922f3b3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 22:50:33.332465 19391 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4575e4961c2048dab05ce7613792355f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 22:50:33.332592 19125 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4575e4961c2048dab05ce7613792355f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 22:50:33.335909 19472 tablet_bootstrap.cc:492] T 4575e4961c2048dab05ce7613792355f P 7beb8031dfff47bd97c0ff3257a0f433: Bootstrap starting.
I20260929 22:50:33.336300 19258 tablet_service.cc:1467] Processing CreateTablet for tablet 4575e4961c2048dab05ce7613792355f (DEFAULT_TABLE table=test-table [id=5a487bbb6bb94c0f9d44ef41d922f3b3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 22:50:33.336601 19258 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4575e4961c2048dab05ce7613792355f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 22:50:33.336608 19472 tablet_bootstrap.cc:654] T 4575e4961c2048dab05ce7613792355f P 7beb8031dfff47bd97c0ff3257a0f433: Neither blocks nor log segments found. Creating new log.
I20260929 22:50:33.337067 19472 log.cc:824] T 4575e4961c2048dab05ce7613792355f P 7beb8031dfff47bd97c0ff3257a0f433: Log is configured to *not* fsync() on all Append() calls
I20260929 22:50:33.337397 19471 tablet_bootstrap.cc:492] T 4575e4961c2048dab05ce7613792355f P 86fcdef103bd41abba87cd27d5ebce1a: Bootstrap starting.
I20260929 22:50:33.337932 19471 tablet_bootstrap.cc:654] T 4575e4961c2048dab05ce7613792355f P 86fcdef103bd41abba87cd27d5ebce1a: Neither blocks nor log segments found. Creating new log.
I20260929 22:50:33.338469 19471 log.cc:824] T 4575e4961c2048dab05ce7613792355f P 86fcdef103bd41abba87cd27d5ebce1a: Log is configured to *not* fsync() on all Append() calls
I20260929 22:50:33.338945 19472 tablet_bootstrap.cc:492] T 4575e4961c2048dab05ce7613792355f P 7beb8031dfff47bd97c0ff3257a0f433: No bootstrap required, opened a new log
I20260929 22:50:33.339018 19472 ts_tablet_manager.cc:1397] T 4575e4961c2048dab05ce7613792355f P 7beb8031dfff47bd97c0ff3257a0f433: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260929 22:50:33.339183 19471 tablet_bootstrap.cc:492] T 4575e4961c2048dab05ce7613792355f P 86fcdef103bd41abba87cd27d5ebce1a: No bootstrap required, opened a new log
I20260929 22:50:33.339247 19471 ts_tablet_manager.cc:1397] T 4575e4961c2048dab05ce7613792355f P 86fcdef103bd41abba87cd27d5ebce1a: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260929 22:50:33.339545 19475 tablet_bootstrap.cc:492] T 4575e4961c2048dab05ce7613792355f P 5938f25392c8486fa687b3fd56847b9a: Bootstrap starting.
I20260929 22:50:33.340096 19475 tablet_bootstrap.cc:654] T 4575e4961c2048dab05ce7613792355f P 5938f25392c8486fa687b3fd56847b9a: Neither blocks nor log segments found. Creating new log.
I20260929 22:50:33.340545 19475 log.cc:824] T 4575e4961c2048dab05ce7613792355f P 5938f25392c8486fa687b3fd56847b9a: Log is configured to *not* fsync() on all Append() calls
I20260929 22:50:33.341153 19475 tablet_bootstrap.cc:492] T 4575e4961c2048dab05ce7613792355f P 5938f25392c8486fa687b3fd56847b9a: No bootstrap required, opened a new log
I20260929 22:50:33.341217 19475 ts_tablet_manager.cc:1397] T 4575e4961c2048dab05ce7613792355f P 5938f25392c8486fa687b3fd56847b9a: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260929 22:50:33.341209 19472 raft_consensus.cc:348] T 4575e4961c2048dab05ce7613792355f P 7beb8031dfff47bd97c0ff3257a0f433 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7beb8031dfff47bd97c0ff3257a0f433" member_type: VOTER last_known_addr { host: "127.14.147.193" port: 43511 } } peers { permanent_uuid: "86fcdef103bd41abba87cd27d5ebce1a" member_type: VOTER last_known_addr { host: "127.14.147.195" port: 33997 } } peers { permanent_uuid: "5938f25392c8486fa687b3fd56847b9a" member_type: VOTER last_known_addr { host: "127.14.147.194" port: 38801 } }
I20260929 22:50:33.341168 19471 raft_consensus.cc:348] T 4575e4961c2048dab05ce7613792355f P 86fcdef103bd41abba87cd27d5ebce1a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7beb8031dfff47bd97c0ff3257a0f433" member_type: VOTER last_known_addr { host: "127.14.147.193" port: 43511 } } peers { permanent_uuid: "86fcdef103bd41abba87cd27d5ebce1a" member_type: VOTER last_known_addr { host: "127.14.147.195" port: 33997 } } peers { permanent_uuid: "5938f25392c8486fa687b3fd56847b9a" member_type: VOTER last_known_addr { host: "127.14.147.194" port: 38801 } }
I20260929 22:50:33.341351 19472 raft_consensus.cc:374] T 4575e4961c2048dab05ce7613792355f P 7beb8031dfff47bd97c0ff3257a0f433 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 22:50:33.341351 19471 raft_consensus.cc:374] T 4575e4961c2048dab05ce7613792355f P 86fcdef103bd41abba87cd27d5ebce1a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 22:50:33.341392 19471 raft_consensus.cc:729] T 4575e4961c2048dab05ce7613792355f P 86fcdef103bd41abba87cd27d5ebce1a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 86fcdef103bd41abba87cd27d5ebce1a, State: Initialized, Role: FOLLOWER
I20260929 22:50:33.341392 19472 raft_consensus.cc:729] T 4575e4961c2048dab05ce7613792355f P 7beb8031dfff47bd97c0ff3257a0f433 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7beb8031dfff47bd97c0ff3257a0f433, State: Initialized, Role: FOLLOWER
I20260929 22:50:33.341529 19472 consensus_queue.cc:260] T 4575e4961c2048dab05ce7613792355f P 7beb8031dfff47bd97c0ff3257a0f433 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7beb8031dfff47bd97c0ff3257a0f433" member_type: VOTER last_known_addr { host: "127.14.147.193" port: 43511 } } peers { permanent_uuid: "86fcdef103bd41abba87cd27d5ebce1a" member_type: VOTER last_known_addr { host: "127.14.147.195" port: 33997 } } peers { permanent_uuid: "5938f25392c8486fa687b3fd56847b9a" member_type: VOTER last_known_addr { host: "127.14.147.194" port: 38801 } }
I20260929 22:50:33.341529 19471 consensus_queue.cc:260] T 4575e4961c2048dab05ce7613792355f P 86fcdef103bd41abba87cd27d5ebce1a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7beb8031dfff47bd97c0ff3257a0f433" member_type: VOTER last_known_addr { host: "127.14.147.193" port: 43511 } } peers { permanent_uuid: "86fcdef103bd41abba87cd27d5ebce1a" member_type: VOTER last_known_addr { host: "127.14.147.195" port: 33997 } } peers { permanent_uuid: "5938f25392c8486fa687b3fd56847b9a" member_type: VOTER last_known_addr { host: "127.14.147.194" port: 38801 } }
I20260929 22:50:33.341804 19471 ts_tablet_manager.cc:1428] T 4575e4961c2048dab05ce7613792355f P 86fcdef103bd41abba87cd27d5ebce1a: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260929 22:50:33.341804 19472 ts_tablet_manager.cc:1428] T 4575e4961c2048dab05ce7613792355f P 7beb8031dfff47bd97c0ff3257a0f433: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260929 22:50:33.341951 19191 heartbeater.cc:503] Master 127.14.147.254:45519 was elected leader, sending a full tablet report...
I20260929 22:50:33.342224 19457 heartbeater.cc:503] Master 127.14.147.254:45519 was elected leader, sending a full tablet report...
I20260929 22:50:33.343065 19475 raft_consensus.cc:348] T 4575e4961c2048dab05ce7613792355f P 5938f25392c8486fa687b3fd56847b9a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7beb8031dfff47bd97c0ff3257a0f433" member_type: VOTER last_known_addr { host: "127.14.147.193" port: 43511 } } peers { permanent_uuid: "86fcdef103bd41abba87cd27d5ebce1a" member_type: VOTER last_known_addr { host: "127.14.147.195" port: 33997 } } peers { permanent_uuid: "5938f25392c8486fa687b3fd56847b9a" member_type: VOTER last_known_addr { host: "127.14.147.194" port: 38801 } }
I20260929 22:50:33.343231 19475 raft_consensus.cc:374] T 4575e4961c2048dab05ce7613792355f P 5938f25392c8486fa687b3fd56847b9a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 22:50:33.343263 19475 raft_consensus.cc:729] T 4575e4961c2048dab05ce7613792355f P 5938f25392c8486fa687b3fd56847b9a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5938f25392c8486fa687b3fd56847b9a, State: Initialized, Role: FOLLOWER
I20260929 22:50:33.343374 19475 consensus_queue.cc:260] T 4575e4961c2048dab05ce7613792355f P 5938f25392c8486fa687b3fd56847b9a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7beb8031dfff47bd97c0ff3257a0f433" member_type: VOTER last_known_addr { host: "127.14.147.193" port: 43511 } } peers { permanent_uuid: "86fcdef103bd41abba87cd27d5ebce1a" member_type: VOTER last_known_addr { host: "127.14.147.195" port: 33997 } } peers { permanent_uuid: "5938f25392c8486fa687b3fd56847b9a" member_type: VOTER last_known_addr { host: "127.14.147.194" port: 38801 } }
I20260929 22:50:33.343621 19475 ts_tablet_manager.cc:1428] T 4575e4961c2048dab05ce7613792355f P 5938f25392c8486fa687b3fd56847b9a: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260929 22:50:33.343659 19324 heartbeater.cc:503] Master 127.14.147.254:45519 was elected leader, sending a full tablet report...
I20260929 22:50:33.366709 19479 raft_consensus.cc:482] T 4575e4961c2048dab05ce7613792355f P 5938f25392c8486fa687b3fd56847b9a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 22:50:33.366842 19479 raft_consensus.cc:504] T 4575e4961c2048dab05ce7613792355f P 5938f25392c8486fa687b3fd56847b9a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7beb8031dfff47bd97c0ff3257a0f433" member_type: VOTER last_known_addr { host: "127.14.147.193" port: 43511 } } peers { permanent_uuid: "86fcdef103bd41abba87cd27d5ebce1a" member_type: VOTER last_known_addr { host: "127.14.147.195" port: 33997 } } peers { permanent_uuid: "5938f25392c8486fa687b3fd56847b9a" member_type: VOTER last_known_addr { host: "127.14.147.194" port: 38801 } }
I20260929 22:50:33.367205 19479 leader_election.cc:288] T 4575e4961c2048dab05ce7613792355f P 5938f25392c8486fa687b3fd56847b9a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7beb8031dfff47bd97c0ff3257a0f433 (127.14.147.193:43511), 86fcdef103bd41abba87cd27d5ebce1a (127.14.147.195:33997)
I20260929 22:50:33.369836 19468 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:50:33.367464 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.147.194:48091 (local address 127.14.147.193:43511)
0929 22:50:33.367522 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0929 22:50:33.367524 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 22:50:33.367535 (+ 11us) server_negotiation.cc:407] Connection header received
0929 22:50:33.367562 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 22:50:33.367563 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 22:50:33.367571 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 22:50:33.367630 (+ 59us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 22:50:33.367776 (+ 146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:50:33.368249 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:50:33.369175 (+ 926us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:50:33.369537 (+ 362us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:50:33.369597 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 22:50:33.369605 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0929 22:50:33.369732 (+ 127us) server_negotiation.cc:299] Negotiation successful
0929 22:50:33.369766 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260929 22:50:33.369836 19326 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:50:33.367366 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.147.193:43511 (local address 127.14.147.194:48091)
0929 22:50:33.367435 (+ 69us) negotiation.cc:107] Waiting for socket to connect
0929 22:50:33.367447 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0929 22:50:33.367494 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 22:50:33.367647 (+ 153us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 22:50:33.367649 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 22:50:33.367651 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 22:50:33.367715 (+ 64us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 22:50:33.367717 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 22:50:33.368325 (+ 608us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 22:50:33.368326 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 22:50:33.369090 (+ 764us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 22:50:33.369092 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 22:50:33.369603 (+ 511us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 22:50:33.369604 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 22:50:33.369674 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 22:50:33.369677 (+ 3us) client_negotiation.cc:769] Sending connection context
0929 22:50:33.369720 (+ 43us) client_negotiation.cc:240] Negotiation successful
0929 22:50:33.369766 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":25}
I20260929 22:50:33.369913 19469 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:50:33.367483 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.147.194:45425 (local address 127.14.147.195:33997)
0929 22:50:33.367529 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0929 22:50:33.367531 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 22:50:33.367682 (+ 151us) server_negotiation.cc:407] Connection header received
0929 22:50:33.367716 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 22:50:33.367717 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 22:50:33.367727 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 22:50:33.367773 (+ 46us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 22:50:33.367912 (+ 139us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:50:33.368376 (+ 464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:50:33.369235 (+ 859us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:50:33.369609 (+ 374us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:50:33.369669 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 22:50:33.369677 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0929 22:50:33.369811 (+ 134us) server_negotiation.cc:299] Negotiation successful
0929 22:50:33.369846 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260929 22:50:33.370131 19480 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:50:33.367367 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.147.195:33997 (local address 127.14.147.194:45425)
0929 22:50:33.367624 (+ 257us) negotiation.cc:107] Waiting for socket to connect
0929 22:50:33.367631 (+ 7us) client_negotiation.cc:174] Beginning negotiation
0929 22:50:33.367673 (+ 42us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 22:50:33.367784 (+ 111us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 22:50:33.367785 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 22:50:33.367787 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 22:50:33.367854 (+ 67us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 22:50:33.367858 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 22:50:33.368440 (+ 582us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 22:50:33.368441 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 22:50:33.369160 (+ 719us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 22:50:33.369161 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 22:50:33.369676 (+ 515us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 22:50:33.369677 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 22:50:33.369746 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 22:50:33.369749 (+ 3us) client_negotiation.cc:769] Sending connection context
0929 22:50:33.369799 (+ 50us) client_negotiation.cc:240] Negotiation successful
0929 22:50:33.369837 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":242,"mutex_wait_us":95,"thread_start_us":108,"threads_started":1}
I20260929 22:50:33.370230 19145 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4575e4961c2048dab05ce7613792355f" candidate_uuid: "5938f25392c8486fa687b3fd56847b9a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7beb8031dfff47bd97c0ff3257a0f433" is_pre_election: true
I20260929 22:50:33.370410 19145 raft_consensus.cc:2454] T 4575e4961c2048dab05ce7613792355f P 7beb8031dfff47bd97c0ff3257a0f433 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5938f25392c8486fa687b3fd56847b9a in term 0.
I20260929 22:50:33.370436 19411 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4575e4961c2048dab05ce7613792355f" candidate_uuid: "5938f25392c8486fa687b3fd56847b9a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "86fcdef103bd41abba87cd27d5ebce1a" is_pre_election: true
I20260929 22:50:33.370594 19411 raft_consensus.cc:2454] T 4575e4961c2048dab05ce7613792355f P 86fcdef103bd41abba87cd27d5ebce1a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5938f25392c8486fa687b3fd56847b9a in term 0.
I20260929 22:50:33.370656 19213 leader_election.cc:302] T 4575e4961c2048dab05ce7613792355f P 5938f25392c8486fa687b3fd56847b9a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5938f25392c8486fa687b3fd56847b9a, 7beb8031dfff47bd97c0ff3257a0f433; no voters:
I20260929 22:50:33.370814 19479 raft_consensus.cc:2781] T 4575e4961c2048dab05ce7613792355f P 5938f25392c8486fa687b3fd56847b9a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 22:50:33.370872 19479 raft_consensus.cc:482] T 4575e4961c2048dab05ce7613792355f P 5938f25392c8486fa687b3fd56847b9a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 22:50:33.370893 19479 raft_consensus.cc:3037] T 4575e4961c2048dab05ce7613792355f P 5938f25392c8486fa687b3fd56847b9a [term 0 FOLLOWER]: Advancing to term 1
I20260929 22:50:33.371614 19479 raft_consensus.cc:504] T 4575e4961c2048dab05ce7613792355f P 5938f25392c8486fa687b3fd56847b9a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7beb8031dfff47bd97c0ff3257a0f433" member_type: VOTER last_known_addr { host: "127.14.147.193" port: 43511 } } peers { permanent_uuid: "86fcdef103bd41abba87cd27d5ebce1a" member_type: VOTER last_known_addr { host: "127.14.147.195" port: 33997 } } peers { permanent_uuid: "5938f25392c8486fa687b3fd56847b9a" member_type: VOTER last_known_addr { host: "127.14.147.194" port: 38801 } }
I20260929 22:50:33.371790 19479 leader_election.cc:288] T 4575e4961c2048dab05ce7613792355f P 5938f25392c8486fa687b3fd56847b9a [CANDIDATE]: Term 1 election: Requested vote from peers 7beb8031dfff47bd97c0ff3257a0f433 (127.14.147.193:43511), 86fcdef103bd41abba87cd27d5ebce1a (127.14.147.195:33997)
I20260929 22:50:33.371924 19411 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4575e4961c2048dab05ce7613792355f" candidate_uuid: "5938f25392c8486fa687b3fd56847b9a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "86fcdef103bd41abba87cd27d5ebce1a"
I20260929 22:50:33.371986 19411 raft_consensus.cc:3037] T 4575e4961c2048dab05ce7613792355f P 86fcdef103bd41abba87cd27d5ebce1a [term 0 FOLLOWER]: Advancing to term 1
I20260929 22:50:33.372834 19411 raft_consensus.cc:2454] T 4575e4961c2048dab05ce7613792355f P 86fcdef103bd41abba87cd27d5ebce1a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5938f25392c8486fa687b3fd56847b9a in term 1.
I20260929 22:50:33.372936 19145 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4575e4961c2048dab05ce7613792355f" candidate_uuid: "5938f25392c8486fa687b3fd56847b9a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7beb8031dfff47bd97c0ff3257a0f433"
I20260929 22:50:33.373013 19145 raft_consensus.cc:3037] T 4575e4961c2048dab05ce7613792355f P 7beb8031dfff47bd97c0ff3257a0f433 [term 0 FOLLOWER]: Advancing to term 1
I20260929 22:50:33.373070 19212 leader_election.cc:302] T 4575e4961c2048dab05ce7613792355f P 5938f25392c8486fa687b3fd56847b9a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5938f25392c8486fa687b3fd56847b9a, 86fcdef103bd41abba87cd27d5ebce1a; no voters:
I20260929 22:50:33.373178 19479 raft_consensus.cc:2781] T 4575e4961c2048dab05ce7613792355f P 5938f25392c8486fa687b3fd56847b9a [term 1 FOLLOWER]: Leader election won for term 1
I20260929 22:50:33.373348 19479 raft_consensus.cc:686] T 4575e4961c2048dab05ce7613792355f P 5938f25392c8486fa687b3fd56847b9a [term 1 LEADER]: Becoming Leader. State: Replica: 5938f25392c8486fa687b3fd56847b9a, State: Running, Role: LEADER
I20260929 22:50:33.373447 19479 consensus_queue.cc:237] T 4575e4961c2048dab05ce7613792355f P 5938f25392c8486fa687b3fd56847b9a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7beb8031dfff47bd97c0ff3257a0f433" member_type: VOTER last_known_addr { host: "127.14.147.193" port: 43511 } } peers { permanent_uuid: "86fcdef103bd41abba87cd27d5ebce1a" member_type: VOTER last_known_addr { host: "127.14.147.195" port: 33997 } } peers { permanent_uuid: "5938f25392c8486fa687b3fd56847b9a" member_type: VOTER last_known_addr { host: "127.14.147.194" port: 38801 } }
I20260929 22:50:33.373868 19145 raft_consensus.cc:2454] T 4575e4961c2048dab05ce7613792355f P 7beb8031dfff47bd97c0ff3257a0f433 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5938f25392c8486fa687b3fd56847b9a in term 1.
I20260929 22:50:33.374274 19000 catalog_manager.cc:5495] T 4575e4961c2048dab05ce7613792355f P 5938f25392c8486fa687b3fd56847b9a reported cstate change: term changed from 0 to 1, leader changed from <none> to 5938f25392c8486fa687b3fd56847b9a (127.14.147.194). New cstate: current_term: 1 leader_uuid: "5938f25392c8486fa687b3fd56847b9a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7beb8031dfff47bd97c0ff3257a0f433" member_type: VOTER last_known_addr { host: "127.14.147.193" port: 43511 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "86fcdef103bd41abba87cd27d5ebce1a" member_type: VOTER last_known_addr { host: "127.14.147.195" port: 33997 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5938f25392c8486fa687b3fd56847b9a" member_type: VOTER last_known_addr { host: "127.14.147.194" port: 38801 } health_report { overall_health: HEALTHY } } }
I20260929 22:50:33.389374 19470 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:50:33.387119 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35884 (local address 127.14.147.194:38801)
0929 22:50:33.387182 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0929 22:50:33.387183 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0929 22:50:33.387191 (+ 8us) server_negotiation.cc:407] Connection header received
0929 22:50:33.387234 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 22:50:33.387234 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 22:50:33.387252 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 22:50:33.387313 (+ 61us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 22:50:33.387448 (+ 135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:50:33.387924 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:50:33.388639 (+ 715us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:50:33.388832 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:50:33.388887 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 22:50:33.389025 (+ 138us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 22:50:33.389129 (+ 104us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 22:50:33.389206 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0929 22:50:33.389243 (+ 37us) server_negotiation.cc:299] Negotiation successful
0929 22:50:33.389265 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260929 22:50:33.391208 19145 raft_consensus.cc:1261] T 4575e4961c2048dab05ce7613792355f P 7beb8031dfff47bd97c0ff3257a0f433 [term 1 FOLLOWER]: Refusing update from remote peer 5938f25392c8486fa687b3fd56847b9a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 22:50:33.391209 19411 raft_consensus.cc:1261] T 4575e4961c2048dab05ce7613792355f P 86fcdef103bd41abba87cd27d5ebce1a [term 1 FOLLOWER]: Refusing update from remote peer 5938f25392c8486fa687b3fd56847b9a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 22:50:33.391556 19481 consensus_queue.cc:1035] T 4575e4961c2048dab05ce7613792355f P 5938f25392c8486fa687b3fd56847b9a [LEADER]: Connected to new peer: Peer: permanent_uuid: "86fcdef103bd41abba87cd27d5ebce1a" member_type: VOTER last_known_addr { host: "127.14.147.195" port: 33997 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 22:50:33.391691 19479 consensus_queue.cc:1035] T 4575e4961c2048dab05ce7613792355f P 5938f25392c8486fa687b3fd56847b9a [LEADER]: Connected to new peer: Peer: permanent_uuid: "7beb8031dfff47bd97c0ff3257a0f433" member_type: VOTER last_known_addr { host: "127.14.147.193" port: 43511 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 22:50:33.394032 19484 mvcc.cc:205] Tried to move back new op lower bound from 7334798267967746048 to 7334798267898273792. Current Snapshot: MvccSnapshot[applied={T|T < 7334798267967746048}]
I20260929 22:50:33.394313 19485 mvcc.cc:205] Tried to move back new op lower bound from 7334798267967746048 to 7334798267898273792. Current Snapshot: MvccSnapshot[applied={T|T < 7334798267967746048}]
I20260929 22:50:33.394766 19486 mvcc.cc:205] Tried to move back new op lower bound from 7334798267967746048 to 7334798267898273792. Current Snapshot: MvccSnapshot[applied={T|T < 7334798267967746048}]
I20260929 22:50:33.396284 19000 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59502:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 22:50:33.396373 19000 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59502:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 22:50:33.397073 19000 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 1f1dfe4cc07b4914b2ae1be28084c996: Sending DeleteTablet for 3 replicas of tablet 4575e4961c2048dab05ce7613792355f
I20260929 22:50:33.397468 19391 tablet_service.cc:1514] Processing DeleteTablet for tablet 4575e4961c2048dab05ce7613792355f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 22:50:33 UTC) from {username='kudu', principal='kudu/127.14.147.254@KRBTEST.COM'} at 127.0.0.1:53582
I20260929 22:50:33.397588 19258 tablet_service.cc:1514] Processing DeleteTablet for tablet 4575e4961c2048dab05ce7613792355f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 22:50:33 UTC) from {username='kudu', principal='kudu/127.14.147.254@KRBTEST.COM'} at 127.0.0.1:35872
I20260929 22:50:33.397706 19125 tablet_service.cc:1514] Processing DeleteTablet for tablet 4575e4961c2048dab05ce7613792355f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 22:50:33 UTC) from {username='kudu', principal='kudu/127.14.147.254@KRBTEST.COM'} at 127.0.0.1:36884
I20260929 22:50:33.397790 19495 tablet_replica.cc:330] T 4575e4961c2048dab05ce7613792355f P 5938f25392c8486fa687b3fd56847b9a: stopping tablet replica
I20260929 22:50:33.397856 19494 tablet_replica.cc:330] T 4575e4961c2048dab05ce7613792355f P 86fcdef103bd41abba87cd27d5ebce1a: stopping tablet replica
I20260929 22:50:33.397886 19495 raft_consensus.cc:2229] T 4575e4961c2048dab05ce7613792355f P 5938f25392c8486fa687b3fd56847b9a [term 1 LEADER]: Raft consensus shutting down.
I20260929 22:50:33.398020 19494 raft_consensus.cc:2229] T 4575e4961c2048dab05ce7613792355f P 86fcdef103bd41abba87cd27d5ebce1a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 22:50:33.398100 19495 raft_consensus.cc:2258] T 4575e4961c2048dab05ce7613792355f P 5938f25392c8486fa687b3fd56847b9a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 22:50:33.398192 14927 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhWOCLk/build/release/bin/kudu with pid 19063
I20260929 22:50:33.398221 19494 raft_consensus.cc:2258] T 4575e4961c2048dab05ce7613792355f P 86fcdef103bd41abba87cd27d5ebce1a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 22:50:33.398568 19495 ts_tablet_manager.cc:1905] T 4575e4961c2048dab05ce7613792355f P 5938f25392c8486fa687b3fd56847b9a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 22:50:33.399832 19494 ts_tablet_manager.cc:1905] T 4575e4961c2048dab05ce7613792355f P 86fcdef103bd41abba87cd27d5ebce1a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 22:50:33.400403 19495 ts_tablet_manager.cc:1918] T 4575e4961c2048dab05ce7613792355f P 5938f25392c8486fa687b3fd56847b9a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 22:50:33.400475 19495 log.cc:1196] T 4575e4961c2048dab05ce7613792355f P 5938f25392c8486fa687b3fd56847b9a: Deleting WAL directory at /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790722213368738-14927-0/minicluster-data/ts-1/wal/wals/4575e4961c2048dab05ce7613792355f
I20260929 22:50:33.400735 19495 ts_tablet_manager.cc:1939] T 4575e4961c2048dab05ce7613792355f P 5938f25392c8486fa687b3fd56847b9a: Deleting consensus metadata
I20260929 22:50:33.401127 18985 catalog_manager.cc:4841] TS 5938f25392c8486fa687b3fd56847b9a (127.14.147.194:38801): tablet 4575e4961c2048dab05ce7613792355f (table test-table [id=5a487bbb6bb94c0f9d44ef41d922f3b3]) successfully deleted
I20260929 22:50:33.401793 19494 ts_tablet_manager.cc:1918] T 4575e4961c2048dab05ce7613792355f P 86fcdef103bd41abba87cd27d5ebce1a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 22:50:33.401930 19494 log.cc:1196] T 4575e4961c2048dab05ce7613792355f P 86fcdef103bd41abba87cd27d5ebce1a: Deleting WAL directory at /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790722213368738-14927-0/minicluster-data/ts-2/wal/wals/4575e4961c2048dab05ce7613792355f
I20260929 22:50:33.402258 19494 ts_tablet_manager.cc:1939] T 4575e4961c2048dab05ce7613792355f P 86fcdef103bd41abba87cd27d5ebce1a: Deleting consensus metadata
I20260929 22:50:33.402729 18986 catalog_manager.cc:4841] TS 86fcdef103bd41abba87cd27d5ebce1a (127.14.147.195:33997): tablet 4575e4961c2048dab05ce7613792355f (table test-table [id=5a487bbb6bb94c0f9d44ef41d922f3b3]) successfully deleted
W20260929 22:50:33.404528 18987 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.14.147.193:43511 (error 108)
I20260929 22:50:33.404742 14927 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhWOCLk/build/release/bin/kudu with pid 19196
I20260929 22:50:33.406396 19466 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:50:33.406271 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.147.193:43511 (local address 127.0.0.1:36894)
0929 22:50:33.406318 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0929 22:50:33.406362 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.147.193:43511: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":22}
W20260929 22:50:33.406520 18987 catalog_manager.cc:4584] TS 7beb8031dfff47bd97c0ff3257a0f433 (127.14.147.193:43511): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 4575e4961c2048dab05ce7613792355f: Network error: Client connection negotiation failed: client connection to 127.14.147.193:43511: connect: Connection refused (error 111)
I20260929 22:50:33.409265 14927 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhWOCLk/build/release/bin/kudu with pid 19329
I20260929 22:50:33.413232 14927 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhWOCLk/build/release/bin/kudu with pid 18970
2026-09-29T22:50:33Z chronyd exiting
[ OK ] SecurityITest.TestDisableWebUI (1819 ms)
[ RUN ] SecurityITest.TestDisableAuthenticationEncryption
2026-09-29T22:50:33Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T22:50:33Z Disabled control of system clock
I20260929 22:50:33.438628 14927 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhWOCLk/build/release/bin/kudu
/tmp/dist-test-taskhWOCLk/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790722213368738-14927-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790722213368738-14927-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790722213368738-14927-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790722213368738-14927-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.147.254:37211
--webserver_interface=127.14.147.254
--webserver_port=0
--builtin_ntp_servers=127.14.147.212:44695
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.147.254:37211
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260929 22:50:33.523485 19501 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 22:50:33.523694 19501 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 22:50:33.523720 19501 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 22:50:33.525030 19501 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 22:50:33.525079 19501 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 22:50:33.525090 19501 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 22:50:33.525101 19501 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 22:50:33.525112 19501 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 22:50:33.526537 19501 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.147.212:44695
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790722213368738-14927-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790722213368738-14927-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.14.147.254:37211
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.147.254:37211
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790722213368738-14927-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.14.147.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19501
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790722213368738-14927-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 21:58:54 UTC on ccycloud.quasar-xguoti.root.comops.site
build id 4681390
I20260929 22:50:33.526775 19501 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 22:50:33.527065 19501 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 22:50:33.531631 19509 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 22:50:33.531638 19506 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 22:50:33.531651 19507 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 22:50:33.531898 19501 server_base.cc:1034] running on GCE node
I20260929 22:50:33.532124 19501 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 22:50:33.532398 19501 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 22:50:33.533550 19501 hybrid_clock.cc:648] HybridClock initialized: now 1790722233533532 us; error 36 us; skew 500 ppm
I20260929 22:50:33.534766 19501 webserver.cc:460] Webserver started at http://127.14.147.254:44163/ using document root <none> and password file <none>
I20260929 22:50:33.535015 19501 fs_manager.cc:359] Metadata directory not provided
I20260929 22:50:33.535063 19501 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 22:50:33.535171 19501 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 22:50:33.536289 19501 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790722213368738-14927-0/minicluster-data/master-0/data/instance:
uuid: "dc3a2ea217c7450ea35c47bbd3d14ecb"
format_stamp: "Formatted at 2026-09-29 22:50:33 on dist-test-slave-98mk"
server_key: "bd05fcfd121303283c628382c9ef733e"
server_key_iv: "94734ce279b575f6b034c4a519bff81d"
server_key_version: "encryptionkey@0"
I20260929 22:50:33.536614 19501 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790722213368738-14927-0/minicluster-data/master-0/wal/instance:
uuid: "dc3a2ea217c7450ea35c47bbd3d14ecb"
format_stamp: "Formatted at 2026-09-29 22:50:33 on dist-test-slave-98mk"
server_key: "bd05fcfd121303283c628382c9ef733e"
server_key_iv: "94734ce279b575f6b034c4a519bff81d"
server_key_version: "encryptionkey@0"
I20260929 22:50:33.537842 19501 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260929 22:50:33.538493 19515 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 22:50:33.538663 19501 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260929 22:50:33.538751 19501 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790722213368738-14927-0/minicluster-data/master-0/data,/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790722213368738-14927-0/minicluster-data/master-0/wal
uuid: "dc3a2ea217c7450ea35c47bbd3d14ecb"
format_stamp: "Formatted at 2026-09-29 22:50:33 on dist-test-slave-98mk"
server_key: "bd05fcfd121303283c628382c9ef733e"
server_key_iv: "94734ce279b575f6b034c4a519bff81d"
server_key_version: "encryptionkey@0"
I20260929 22:50:33.538810 19501 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790722213368738-14927-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790722213368738-14927-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790722213368738-14927-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 22:50:33.542027 19501 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 22:50:33.542250 19501 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 22:50:33.542342 19501 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 22:50:33.545758 19501 rpc_server.cc:307] RPC server started. Bound to: 127.14.147.254:37211
I20260929 22:50:33.545792 19567 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.147.254:37211 every 8 connection(s)
I20260929 22:50:33.546078 19501 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790722213368738-14927-0/minicluster-data/master-0/data/info.pb
I20260929 22:50:33.546726 19568 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 22:50:33.549436 19568 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dc3a2ea217c7450ea35c47bbd3d14ecb: Bootstrap starting.
I20260929 22:50:33.549965 19568 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P dc3a2ea217c7450ea35c47bbd3d14ecb: Neither blocks nor log segments found. Creating new log.
I20260929 22:50:33.550237 19568 log.cc:824] T 00000000000000000000000000000000 P dc3a2ea217c7450ea35c47bbd3d14ecb: Log is configured to *not* fsync() on all Append() calls
I20260929 22:50:33.550838 19568 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dc3a2ea217c7450ea35c47bbd3d14ecb: No bootstrap required, opened a new log
I20260929 22:50:33.552417 19568 raft_consensus.cc:348] T 00000000000000000000000000000000 P dc3a2ea217c7450ea35c47bbd3d14ecb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dc3a2ea217c7450ea35c47bbd3d14ecb" member_type: VOTER last_known_addr { host: "127.14.147.254" port: 37211 } }
I20260929 22:50:33.552552 19568 raft_consensus.cc:374] T 00000000000000000000000000000000 P dc3a2ea217c7450ea35c47bbd3d14ecb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 22:50:33.552572 19568 raft_consensus.cc:729] T 00000000000000000000000000000000 P dc3a2ea217c7450ea35c47bbd3d14ecb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dc3a2ea217c7450ea35c47bbd3d14ecb, State: Initialized, Role: FOLLOWER
I20260929 22:50:33.552711 19568 consensus_queue.cc:260] T 00000000000000000000000000000000 P dc3a2ea217c7450ea35c47bbd3d14ecb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dc3a2ea217c7450ea35c47bbd3d14ecb" member_type: VOTER last_known_addr { host: "127.14.147.254" port: 37211 } }
I20260929 22:50:33.552772 19568 raft_consensus.cc:388] T 00000000000000000000000000000000 P dc3a2ea217c7450ea35c47bbd3d14ecb [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 22:50:33.552796 19568 raft_consensus.cc:482] T 00000000000000000000000000000000 P dc3a2ea217c7450ea35c47bbd3d14ecb [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 22:50:33.552842 19568 raft_consensus.cc:3037] T 00000000000000000000000000000000 P dc3a2ea217c7450ea35c47bbd3d14ecb [term 0 FOLLOWER]: Advancing to term 1
I20260929 22:50:33.552866 14927 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhWOCLk/build/release/bin/kudu as pid 19501
I20260929 22:50:33.552968 14927 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790722213368738-14927-0/minicluster-data/master-0/wal/instance
I20260929 22:50:33.553134 14927 external_mini_cluster.cc:1428] Setting key 972fd6d7383929021648a9a8e3c55914
I20260929 22:50:33.553462 19568 raft_consensus.cc:504] T 00000000000000000000000000000000 P dc3a2ea217c7450ea35c47bbd3d14ecb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dc3a2ea217c7450ea35c47bbd3d14ecb" member_type: VOTER last_known_addr { host: "127.14.147.254" port: 37211 } }
I20260929 22:50:33.553609 19568 leader_election.cc:302] T 00000000000000000000000000000000 P dc3a2ea217c7450ea35c47bbd3d14ecb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: dc3a2ea217c7450ea35c47bbd3d14ecb; no voters:
I20260929 22:50:33.553761 19568 leader_election.cc:288] T 00000000000000000000000000000000 P dc3a2ea217c7450ea35c47bbd3d14ecb [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 22:50:33.553791 19573 raft_consensus.cc:2781] T 00000000000000000000000000000000 P dc3a2ea217c7450ea35c47bbd3d14ecb [term 1 FOLLOWER]: Leader election won for term 1
I20260929 22:50:33.553922 19573 raft_consensus.cc:686] T 00000000000000000000000000000000 P dc3a2ea217c7450ea35c47bbd3d14ecb [term 1 LEADER]: Becoming Leader. State: Replica: dc3a2ea217c7450ea35c47bbd3d14ecb, State: Running, Role: LEADER
I20260929 22:50:33.554049 19568 sys_catalog.cc:564] T 00000000000000000000000000000000 P dc3a2ea217c7450ea35c47bbd3d14ecb [sys.catalog]: configured and running, proceeding with master startup.
I20260929 22:50:33.554046 19573 consensus_queue.cc:237] T 00000000000000000000000000000000 P dc3a2ea217c7450ea35c47bbd3d14ecb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dc3a2ea217c7450ea35c47bbd3d14ecb" member_type: VOTER last_known_addr { host: "127.14.147.254" port: 37211 } }
I20260929 22:50:33.554631 19574 sys_catalog.cc:455] T 00000000000000000000000000000000 P dc3a2ea217c7450ea35c47bbd3d14ecb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "dc3a2ea217c7450ea35c47bbd3d14ecb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dc3a2ea217c7450ea35c47bbd3d14ecb" member_type: VOTER last_known_addr { host: "127.14.147.254" port: 37211 } } }
I20260929 22:50:33.554708 19574 sys_catalog.cc:458] T 00000000000000000000000000000000 P dc3a2ea217c7450ea35c47bbd3d14ecb [sys.catalog]: This master's current role is: LEADER
I20260929 22:50:33.555020 19578 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 22:50:33.555591 19572 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:50:33.553593 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42622 (local address 127.14.147.254:37211)
0929 22:50:33.553821 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0929 22:50:33.553830 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 22:50:33.553865 (+ 35us) server_negotiation.cc:407] Connection header received
0929 22:50:33.553982 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 22:50:33.553989 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 22:50:33.554033 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 22:50:33.554445 (+ 412us) server_negotiation.cc:226] Negotiated authn=SASL
0929 22:50:33.554979 (+ 534us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 22:50:33.554983 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 22:50:33.554983 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 22:50:33.554987 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0929 22:50:33.555026 (+ 39us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 22:50:33.555040 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 22:50:33.555245 (+ 205us) server_negotiation.cc:1035] Waiting for connection context
0929 22:50:33.555278 (+ 33us) server_negotiation.cc:299] Negotiation successful
0929 22:50:33.555342 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":141,"thread_start_us":83,"threads_started":1}
I20260929 22:50:33.555661 19578 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 22:50:33.555025 19575 sys_catalog.cc:455] T 00000000000000000000000000000000 P dc3a2ea217c7450ea35c47bbd3d14ecb [sys.catalog]: SysCatalogTable state changed. Reason: New leader dc3a2ea217c7450ea35c47bbd3d14ecb. Latest consensus state: current_term: 1 leader_uuid: "dc3a2ea217c7450ea35c47bbd3d14ecb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dc3a2ea217c7450ea35c47bbd3d14ecb" member_type: VOTER last_known_addr { host: "127.14.147.254" port: 37211 } } }
I20260929 22:50:33.556349 19575 sys_catalog.cc:458] T 00000000000000000000000000000000 P dc3a2ea217c7450ea35c47bbd3d14ecb [sys.catalog]: This master's current role is: LEADER
I20260929 22:50:33.557601 19578 catalog_manager.cc:1347] Generated new cluster ID: 0a84f6eff0f54b90b9d60598ddcb4e80
I20260929 22:50:33.557657 19578 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 22:50:33.582439 19578 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 22:50:33.582628 19578 catalog_manager.cc:1504] Loading token signing keys...
I20260929 22:50:33.589460 19578 catalog_manager.cc:5868] T 00000000000000000000000000000000 P dc3a2ea217c7450ea35c47bbd3d14ecb: Generated new TSK 0
I20260929 22:50:33.589664 19578 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 22:50:33.599123 14927 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhWOCLk/build/release/bin/kudu
/tmp/dist-test-taskhWOCLk/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790722213368738-14927-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790722213368738-14927-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790722213368738-14927-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790722213368738-14927-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.147.193:0
--local_ip_for_outbound_sockets=127.14.147.193
--webserver_interface=127.14.147.193
--webserver_port=0
--tserver_master_addrs=127.14.147.254:37211
--builtin_ntp_servers=127.14.147.212:44695
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260929 22:50:33.688442 19592 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 22:50:33.688650 19592 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 22:50:33.688674 19592 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 22:50:33.689900 19592 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 22:50:33.689949 19592 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 22:50:33.689976 19592 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.147.193
I20260929 22:50:33.691444 19592 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.147.212:44695
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790722213368738-14927-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790722213368738-14927-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.147.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790722213368738-14927-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.14.147.193
--webserver_port=0
--tserver_master_addrs=127.14.147.254:37211
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19592
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.147.193
--log_dir=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790722213368738-14927-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 21:58:54 UTC on ccycloud.quasar-xguoti.root.comops.site
build id 4681390
I20260929 22:50:33.691679 19592 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 22:50:33.691967 19592 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 22:50:33.697158 19598 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 22:50:33.697270 19592 server_base.cc:1034] running on GCE node
W20260929 22:50:33.697319 19597 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 22:50:33.697180 19600 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 22:50:33.697594 19592 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 22:50:33.697846 19592 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 22:50:33.699005 19592 hybrid_clock.cc:648] HybridClock initialized: now 1790722233698992 us; error 30 us; skew 500 ppm
I20260929 22:50:33.700313 19592 webserver.cc:460] Webserver started at http://127.14.147.193:37521/ using document root <none> and password file <none>
I20260929 22:50:33.700549 19592 fs_manager.cc:359] Metadata directory not provided
I20260929 22:50:33.700618 19592 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 22:50:33.700754 19592 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 22:50:33.701821 19592 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790722213368738-14927-0/minicluster-data/ts-0/data/instance:
uuid: "184753f0d08d4a23bc3c0fc73380a0e5"
format_stamp: "Formatted at 2026-09-29 22:50:33 on dist-test-slave-98mk"
server_key: "0215f328145e282e1062f558ebe0395a"
server_key_iv: "031cd74ee796eff6ef3e27b647b12256"
server_key_version: "encryptionkey@0"
I20260929 22:50:33.702150 19592 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790722213368738-14927-0/minicluster-data/ts-0/wal/instance:
uuid: "184753f0d08d4a23bc3c0fc73380a0e5"
format_stamp: "Formatted at 2026-09-29 22:50:33 on dist-test-slave-98mk"
server_key: "0215f328145e282e1062f558ebe0395a"
server_key_iv: "031cd74ee796eff6ef3e27b647b12256"
server_key_version: "encryptionkey@0"
I20260929 22:50:33.703536 19592 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260929 22:50:33.704273 19606 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 22:50:33.704418 19592 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260929 22:50:33.704499 19592 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790722213368738-14927-0/minicluster-data/ts-0/data,/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790722213368738-14927-0/minicluster-data/ts-0/wal
uuid: "184753f0d08d4a23bc3c0fc73380a0e5"
format_stamp: "Formatted at 2026-09-29 22:50:33 on dist-test-slave-98mk"
server_key: "0215f328145e282e1062f558ebe0395a"
server_key_iv: "031cd74ee796eff6ef3e27b647b12256"
server_key_version: "encryptionkey@0"
I20260929 22:50:33.704564 19592 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790722213368738-14927-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790722213368738-14927-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790722213368738-14927-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 22:50:33.707772 19592 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 22:50:33.708024 19592 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 22:50:33.708137 19592 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 22:50:33.708320 19592 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 22:50:33.708592 19592 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 22:50:33.708622 19592 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 22:50:33.708648 19592 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 22:50:33.708671 19592 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 22:50:33.713966 19592 rpc_server.cc:307] RPC server started. Bound to: 127.14.147.193:39067
I20260929 22:50:33.714027 19719 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.147.193:39067 every 8 connection(s)
I20260929 22:50:33.714324 19592 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790722213368738-14927-0/minicluster-data/ts-0/data/info.pb
I20260929 22:50:33.717290 19572 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:50:33.715009 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.147.193:40411 (local address 127.14.147.254:37211)
0929 22:50:33.715053 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0929 22:50:33.715055 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 22:50:33.715271 (+ 216us) server_negotiation.cc:407] Connection header received
0929 22:50:33.715813 (+ 542us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 22:50:33.715815 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 22:50:33.715824 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 22:50:33.715884 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0929 22:50:33.717044 (+ 1160us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 22:50:33.717045 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 22:50:33.717046 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 22:50:33.717051 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0929 22:50:33.717068 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 22:50:33.717073 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 22:50:33.717131 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0929 22:50:33.717200 (+ 69us) server_negotiation.cc:299] Negotiation successful
0929 22:50:33.717215 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260929 22:50:33.717479 19722 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:50:33.714992 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.147.254:37211 (local address 127.14.147.193:40411)
0929 22:50:33.715154 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0929 22:50:33.715188 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0929 22:50:33.715717 (+ 529us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 22:50:33.715959 (+ 242us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 22:50:33.715964 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 22:50:33.715973 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
0929 22:50:33.716892 (+ 919us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 22:50:33.716892 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0929 22:50:33.716923 (+ 31us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 22:50:33.716954 (+ 31us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 22:50:33.717129 (+ 175us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 22:50:33.717134 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 22:50:33.717137 (+ 3us) client_negotiation.cc:769] Sending connection context
0929 22:50:33.717208 (+ 71us) client_negotiation.cc:240] Negotiation successful
0929 22:50:33.717272 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":102,"thread_start_us":68,"threads_started":1}
I20260929 22:50:33.717903 19720 heartbeater.cc:347] Connected to a master server at 127.14.147.254:37211
I20260929 22:50:33.717993 19720 heartbeater.cc:464] Registering TS with master...
I20260929 22:50:33.718144 19720 heartbeater.cc:511] Master 127.14.147.254:37211 requested a full tablet report, sending...
I20260929 22:50:33.718621 19532 ts_manager.cc:194] Registered new tserver with Master: 184753f0d08d4a23bc3c0fc73380a0e5 (127.14.147.193:39067)
I20260929 22:50:33.724591 14927 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhWOCLk/build/release/bin/kudu as pid 19592
I20260929 22:50:33.724690 14927 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790722213368738-14927-0/minicluster-data/ts-0/wal/instance
I20260929 22:50:33.724829 14927 external_mini_cluster.cc:1428] Setting key 283fd9023e7402043a48df72c1ca1370
I20260929 22:50:33.725773 14927 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhWOCLk/build/release/bin/kudu
/tmp/dist-test-taskhWOCLk/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790722213368738-14927-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790722213368738-14927-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790722213368738-14927-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790722213368738-14927-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.147.194:0
--local_ip_for_outbound_sockets=127.14.147.194
--webserver_interface=127.14.147.194
--webserver_port=0
--tserver_master_addrs=127.14.147.254:37211
--builtin_ntp_servers=127.14.147.212:44695
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260929 22:50:33.804623 19723 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 22:50:33.804841 19723 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 22:50:33.804872 19723 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 22:50:33.806078 19723 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 22:50:33.806123 19723 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 22:50:33.806157 19723 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.147.194
I20260929 22:50:33.807627 19723 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.147.212:44695
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790722213368738-14927-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790722213368738-14927-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.147.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790722213368738-14927-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.14.147.194
--webserver_port=0
--tserver_master_addrs=127.14.147.254:37211
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19723
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.147.194
--log_dir=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790722213368738-14927-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 21:58:54 UTC on ccycloud.quasar-xguoti.root.comops.site
build id 4681390
I20260929 22:50:33.807868 19723 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 22:50:33.808166 19723 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 22:50:33.813208 19728 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 22:50:33.813207 19731 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 22:50:33.813207 19729 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 22:50:33.813225 19723 server_base.cc:1034] running on GCE node
I20260929 22:50:33.813868 19723 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 22:50:33.814080 19723 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 22:50:33.815218 19723 hybrid_clock.cc:648] HybridClock initialized: now 1790722233815203 us; error 31 us; skew 500 ppm
I20260929 22:50:33.816475 19723 webserver.cc:460] Webserver started at http://127.14.147.194:39277/ using document root <none> and password file <none>
I20260929 22:50:33.816696 19723 fs_manager.cc:359] Metadata directory not provided
I20260929 22:50:33.816746 19723 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 22:50:33.816874 19723 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 22:50:33.817956 19723 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790722213368738-14927-0/minicluster-data/ts-1/data/instance:
uuid: "7343d1ca698944e09c510ec0227e7322"
format_stamp: "Formatted at 2026-09-29 22:50:33 on dist-test-slave-98mk"
server_key: "6e09c9aa4b045435a831ba4100eed83d"
server_key_iv: "ee9efe773aa2e4a3a87ce54f9906f42b"
server_key_version: "encryptionkey@0"
I20260929 22:50:33.818297 19723 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790722213368738-14927-0/minicluster-data/ts-1/wal/instance:
uuid: "7343d1ca698944e09c510ec0227e7322"
format_stamp: "Formatted at 2026-09-29 22:50:33 on dist-test-slave-98mk"
server_key: "6e09c9aa4b045435a831ba4100eed83d"
server_key_iv: "ee9efe773aa2e4a3a87ce54f9906f42b"
server_key_version: "encryptionkey@0"
I20260929 22:50:33.819651 19723 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260929 22:50:33.820430 19737 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 22:50:33.820583 19723 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260929 22:50:33.820664 19723 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790722213368738-14927-0/minicluster-data/ts-1/data,/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790722213368738-14927-0/minicluster-data/ts-1/wal
uuid: "7343d1ca698944e09c510ec0227e7322"
format_stamp: "Formatted at 2026-09-29 22:50:33 on dist-test-slave-98mk"
server_key: "6e09c9aa4b045435a831ba4100eed83d"
server_key_iv: "ee9efe773aa2e4a3a87ce54f9906f42b"
server_key_version: "encryptionkey@0"
I20260929 22:50:33.820721 19723 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790722213368738-14927-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790722213368738-14927-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790722213368738-14927-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 22:50:33.823972 19723 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 22:50:33.824200 19723 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 22:50:33.824292 19723 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 22:50:33.824463 19723 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 22:50:33.824712 19723 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 22:50:33.824738 19723 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 22:50:33.824764 19723 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 22:50:33.824781 19723 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 22:50:33.830483 19723 rpc_server.cc:307] RPC server started. Bound to: 127.14.147.194:42977
I20260929 22:50:33.830549 19850 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.147.194:42977 every 8 connection(s)
I20260929 22:50:33.830860 19723 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790722213368738-14927-0/minicluster-data/ts-1/data/info.pb
I20260929 22:50:33.833765 19572 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:50:33.831576 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.147.194:43019 (local address 127.14.147.254:37211)
0929 22:50:33.831683 (+ 107us) server_negotiation.cc:206] Beginning negotiation
0929 22:50:33.831685 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 22:50:33.831912 (+ 227us) server_negotiation.cc:407] Connection header received
0929 22:50:33.832412 (+ 500us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 22:50:33.832414 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 22:50:33.832420 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 22:50:33.832497 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0929 22:50:33.833573 (+ 1076us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 22:50:33.833574 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 22:50:33.833574 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 22:50:33.833578 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0929 22:50:33.833593 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 22:50:33.833598 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 22:50:33.833641 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0929 22:50:33.833697 (+ 56us) server_negotiation.cc:299] Negotiation successful
0929 22:50:33.833709 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260929 22:50:33.833984 19853 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:50:33.831596 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.147.254:37211 (local address 127.14.147.194:43019)
0929 22:50:33.831781 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0929 22:50:33.831823 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0929 22:50:33.832306 (+ 483us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 22:50:33.832554 (+ 248us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 22:50:33.832557 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 22:50:33.832565 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0929 22:50:33.833424 (+ 859us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 22:50:33.833424 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0929 22:50:33.833454 (+ 30us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 22:50:33.833486 (+ 32us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 22:50:33.833644 (+ 158us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 22:50:33.833646 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 22:50:33.833649 (+ 3us) client_negotiation.cc:769] Sending connection context
0929 22:50:33.833698 (+ 49us) client_negotiation.cc:240] Negotiation successful
0929 22:50:33.833754 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":79,"threads_started":1}
I20260929 22:50:33.834441 19851 heartbeater.cc:347] Connected to a master server at 127.14.147.254:37211
I20260929 22:50:33.834518 19851 heartbeater.cc:464] Registering TS with master...
I20260929 22:50:33.834657 19851 heartbeater.cc:511] Master 127.14.147.254:37211 requested a full tablet report, sending...
I20260929 22:50:33.834977 19532 ts_manager.cc:194] Registered new tserver with Master: 7343d1ca698944e09c510ec0227e7322 (127.14.147.194:42977)
I20260929 22:50:33.840543 14927 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhWOCLk/build/release/bin/kudu as pid 19723
I20260929 22:50:33.840638 14927 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790722213368738-14927-0/minicluster-data/ts-1/wal/instance
I20260929 22:50:33.840818 14927 external_mini_cluster.cc:1428] Setting key 4423e380612e7e1f821b906b2ac4f217
I20260929 22:50:33.841765 14927 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhWOCLk/build/release/bin/kudu
/tmp/dist-test-taskhWOCLk/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790722213368738-14927-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790722213368738-14927-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790722213368738-14927-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790722213368738-14927-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.147.195:0
--local_ip_for_outbound_sockets=127.14.147.195
--webserver_interface=127.14.147.195
--webserver_port=0
--tserver_master_addrs=127.14.147.254:37211
--builtin_ntp_servers=127.14.147.212:44695
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260929 22:50:33.922461 19854 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 22:50:33.922679 19854 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 22:50:33.922703 19854 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 22:50:33.924002 19854 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 22:50:33.924050 19854 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 22:50:33.924077 19854 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.147.195
I20260929 22:50:33.925554 19854 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.147.212:44695
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790722213368738-14927-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790722213368738-14927-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.147.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790722213368738-14927-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.14.147.195
--webserver_port=0
--tserver_master_addrs=127.14.147.254:37211
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19854
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.147.195
--log_dir=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790722213368738-14927-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 21:58:54 UTC on ccycloud.quasar-xguoti.root.comops.site
build id 4681390
I20260929 22:50:33.925818 19854 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 22:50:33.926116 19854 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 22:50:33.930941 19859 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 22:50:33.930913 19862 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 22:50:33.930918 19860 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 22:50:33.931099 19854 server_base.cc:1034] running on GCE node
I20260929 22:50:33.931388 19854 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 22:50:33.931680 19854 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 22:50:33.932869 19854 hybrid_clock.cc:648] HybridClock initialized: now 1790722233932848 us; error 30 us; skew 500 ppm
I20260929 22:50:33.934171 19854 webserver.cc:460] Webserver started at http://127.14.147.195:40153/ using document root <none> and password file <none>
I20260929 22:50:33.934381 19854 fs_manager.cc:359] Metadata directory not provided
I20260929 22:50:33.934422 19854 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 22:50:33.934515 19854 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 22:50:33.935974 19854 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790722213368738-14927-0/minicluster-data/ts-2/data/instance:
uuid: "11cdc23f46eb4e20bec6d48401a1758c"
format_stamp: "Formatted at 2026-09-29 22:50:33 on dist-test-slave-98mk"
server_key: "509b1307a44779f82c9ba19f51ea5193"
server_key_iv: "3b85517473b22ae12f5dd9326e60f9a8"
server_key_version: "encryptionkey@0"
I20260929 22:50:33.936365 19854 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790722213368738-14927-0/minicluster-data/ts-2/wal/instance:
uuid: "11cdc23f46eb4e20bec6d48401a1758c"
format_stamp: "Formatted at 2026-09-29 22:50:33 on dist-test-slave-98mk"
server_key: "509b1307a44779f82c9ba19f51ea5193"
server_key_iv: "3b85517473b22ae12f5dd9326e60f9a8"
server_key_version: "encryptionkey@0"
I20260929 22:50:33.938011 19854 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260929 22:50:33.938755 19868 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 22:50:33.938894 19854 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260929 22:50:33.938973 19854 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790722213368738-14927-0/minicluster-data/ts-2/data,/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790722213368738-14927-0/minicluster-data/ts-2/wal
uuid: "11cdc23f46eb4e20bec6d48401a1758c"
format_stamp: "Formatted at 2026-09-29 22:50:33 on dist-test-slave-98mk"
server_key: "509b1307a44779f82c9ba19f51ea5193"
server_key_iv: "3b85517473b22ae12f5dd9326e60f9a8"
server_key_version: "encryptionkey@0"
I20260929 22:50:33.939043 19854 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790722213368738-14927-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790722213368738-14927-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790722213368738-14927-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 22:50:33.942473 19854 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 22:50:33.942718 19854 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 22:50:33.942830 19854 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 22:50:33.943006 19854 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 22:50:33.943290 19854 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 22:50:33.943321 19854 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 22:50:33.943348 19854 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 22:50:33.943364 19854 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 22:50:33.948880 19854 rpc_server.cc:307] RPC server started. Bound to: 127.14.147.195:42791
I20260929 22:50:33.948928 19981 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.147.195:42791 every 8 connection(s)
I20260929 22:50:33.949250 19854 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790722213368738-14927-0/minicluster-data/ts-2/data/info.pb
I20260929 22:50:33.952122 19572 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:50:33.949948 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.147.195:34575 (local address 127.14.147.254:37211)
0929 22:50:33.949999 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0929 22:50:33.950000 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0929 22:50:33.950235 (+ 235us) server_negotiation.cc:407] Connection header received
0929 22:50:33.950705 (+ 470us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 22:50:33.950706 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 22:50:33.950712 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 22:50:33.950764 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0929 22:50:33.951883 (+ 1119us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 22:50:33.951885 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 22:50:33.951886 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 22:50:33.951890 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0929 22:50:33.951905 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 22:50:33.951909 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 22:50:33.951960 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0929 22:50:33.952033 (+ 73us) server_negotiation.cc:299] Negotiation successful
0929 22:50:33.952049 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260929 22:50:33.952319 19984 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:50:33.949919 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.147.254:37211 (local address 127.14.147.195:34575)
0929 22:50:33.950101 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0929 22:50:33.950155 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0929 22:50:33.950613 (+ 458us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 22:50:33.950830 (+ 217us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 22:50:33.950833 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 22:50:33.950842 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
0929 22:50:33.951741 (+ 899us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 22:50:33.951742 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0929 22:50:33.951770 (+ 28us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 22:50:33.951799 (+ 29us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 22:50:33.951964 (+ 165us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 22:50:33.951969 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 22:50:33.951973 (+ 4us) client_negotiation.cc:769] Sending connection context
0929 22:50:33.952040 (+ 67us) client_negotiation.cc:240] Negotiation successful
0929 22:50:33.952099 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":84,"threads_started":1}
I20260929 22:50:33.952854 19982 heartbeater.cc:347] Connected to a master server at 127.14.147.254:37211
I20260929 22:50:33.952943 19982 heartbeater.cc:464] Registering TS with master...
I20260929 22:50:33.953060 19982 heartbeater.cc:511] Master 127.14.147.254:37211 requested a full tablet report, sending...
I20260929 22:50:33.953397 19532 ts_manager.cc:194] Registered new tserver with Master: 11cdc23f46eb4e20bec6d48401a1758c (127.14.147.195:42791)
I20260929 22:50:33.956470 14927 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhWOCLk/build/release/bin/kudu as pid 19854
I20260929 22:50:33.956557 14927 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790722213368738-14927-0/minicluster-data/ts-2/wal/instance
I20260929 22:50:33.956735 14927 external_mini_cluster.cc:1428] Setting key 7ab1392d8e6d53d206b18bb57bc07bb9
I20260929 22:50:33.957520 14927 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 22:50:33.959833 19572 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:50:33.958748 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52866 (local address 127.14.147.254:37211)
0929 22:50:33.958788 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0929 22:50:33.958790 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 22:50:33.958930 (+ 140us) server_negotiation.cc:407] Connection header received
0929 22:50:33.958999 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 22:50:33.959000 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 22:50:33.959004 (+ 4us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 22:50:33.959044 (+ 40us) server_negotiation.cc:226] Negotiated authn=SASL
0929 22:50:33.959627 (+ 583us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 22:50:33.959628 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 22:50:33.959628 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 22:50:33.959633 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0929 22:50:33.959649 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 22:50:33.959654 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 22:50:33.959700 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0929 22:50:33.959746 (+ 46us) server_negotiation.cc:299] Negotiation successful
0929 22:50:33.959761 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260929 22:50:33.961143 19532 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:52866:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 22:50:33.962671 19532 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 22:50:33.966194 19990 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:50:33.964629 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.147.195:42791 (local address 127.0.0.1:51362)
0929 22:50:33.964910 (+ 281us) negotiation.cc:107] Waiting for socket to connect
0929 22:50:33.964952 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0929 22:50:33.965009 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 22:50:33.965290 (+ 281us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 22:50:33.965297 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 22:50:33.965305 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0929 22:50:33.965776 (+ 471us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 22:50:33.965777 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0929 22:50:33.965792 (+ 15us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 22:50:33.965816 (+ 24us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 22:50:33.966053 (+ 237us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 22:50:33.966058 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 22:50:33.966061 (+ 3us) client_negotiation.cc:769] Sending connection context
0929 22:50:33.966107 (+ 46us) client_negotiation.cc:240] Negotiation successful
0929 22:50:33.966119 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":252,"thread_start_us":55,"threads_started":1}
I20260929 22:50:33.966267 19995 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:50:33.965014 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39922 (local address 127.14.147.193:39067)
0929 22:50:33.965166 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0929 22:50:33.965170 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 22:50:33.965178 (+ 8us) server_negotiation.cc:407] Connection header received
0929 22:50:33.965202 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 22:50:33.965204 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 22:50:33.965211 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 22:50:33.965683 (+ 472us) server_negotiation.cc:226] Negotiated authn=SASL
0929 22:50:33.965960 (+ 277us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 22:50:33.965963 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 22:50:33.965963 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 22:50:33.965967 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0929 22:50:33.965990 (+ 23us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 22:50:33.966005 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 22:50:33.966159 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0929 22:50:33.966180 (+ 21us) server_negotiation.cc:299] Negotiation successful
0929 22:50:33.966193 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":115,"thread_start_us":80,"threads_started":1}
I20260929 22:50:33.966495 19994 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:50:33.964826 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.147.193:39067 (local address 127.0.0.1:39922)
0929 22:50:33.965036 (+ 210us) negotiation.cc:107] Waiting for socket to connect
0929 22:50:33.965046 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0929 22:50:33.965091 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 22:50:33.965309 (+ 218us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 22:50:33.965310 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 22:50:33.965313 (+ 3us) client_negotiation.cc:189] Negotiated authn=SASL
0929 22:50:33.965879 (+ 566us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 22:50:33.965880 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0929 22:50:33.965886 (+ 6us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 22:50:33.965898 (+ 12us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 22:50:33.966074 (+ 176us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 22:50:33.966075 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 22:50:33.966075 (+ 0us) client_negotiation.cc:769] Sending connection context
0929 22:50:33.966106 (+ 31us) client_negotiation.cc:240] Negotiation successful
0929 22:50:33.966121 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":191,"mutex_wait_us":1,"thread_start_us":73,"threads_started":1}
I20260929 22:50:33.966578 19991 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:50:33.964670 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.147.194:42977 (local address 127.0.0.1:41822)
0929 22:50:33.964910 (+ 240us) negotiation.cc:107] Waiting for socket to connect
0929 22:50:33.964952 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0929 22:50:33.965009 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 22:50:33.965881 (+ 872us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 22:50:33.965882 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 22:50:33.965885 (+ 3us) client_negotiation.cc:189] Negotiated authn=SASL
0929 22:50:33.966196 (+ 311us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 22:50:33.966197 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0929 22:50:33.966203 (+ 6us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 22:50:33.966213 (+ 10us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 22:50:33.966431 (+ 218us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 22:50:33.966432 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 22:50:33.966433 (+ 1us) client_negotiation.cc:769] Sending connection context
0929 22:50:33.966470 (+ 37us) client_negotiation.cc:240] Negotiation successful
0929 22:50:33.966483 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":224,"thread_start_us":109,"threads_started":1}
I20260929 22:50:33.966617 19993 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:50:33.964785 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41822 (local address 127.14.147.194:42977)
0929 22:50:33.965198 (+ 413us) server_negotiation.cc:206] Beginning negotiation
0929 22:50:33.965203 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 22:50:33.965212 (+ 9us) server_negotiation.cc:407] Connection header received
0929 22:50:33.965234 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 22:50:33.965235 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 22:50:33.965244 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 22:50:33.965311 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0929 22:50:33.966292 (+ 981us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 22:50:33.966299 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 22:50:33.966299 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 22:50:33.966304 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0929 22:50:33.966339 (+ 35us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 22:50:33.966360 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 22:50:33.966409 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0929 22:50:33.966530 (+ 121us) server_negotiation.cc:299] Negotiation successful
0929 22:50:33.966545 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":376,"thread_start_us":78,"threads_started":1}
I20260929 22:50:33.966958 19992 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:50:33.964743 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51362 (local address 127.14.147.195:42791)
0929 22:50:33.965127 (+ 384us) server_negotiation.cc:206] Beginning negotiation
0929 22:50:33.965131 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 22:50:33.965141 (+ 10us) server_negotiation.cc:407] Connection header received
0929 22:50:33.965167 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 22:50:33.965170 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 22:50:33.965180 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 22:50:33.965265 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0929 22:50:33.965900 (+ 635us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 22:50:33.965909 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 22:50:33.965909 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 22:50:33.965913 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0929 22:50:33.965949 (+ 36us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 22:50:33.965978 (+ 29us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 22:50:33.966839 (+ 861us) server_negotiation.cc:1035] Waiting for connection context
0929 22:50:33.966862 (+ 23us) server_negotiation.cc:299] Negotiation successful
0929 22:50:33.966878 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":344,"thread_start_us":68,"threads_started":1}
I20260929 22:50:33.967188 19654 tablet_service.cc:1467] Processing CreateTablet for tablet a43f88ce0a244c838c7340b5e779965f (DEFAULT_TABLE table=test-table [id=b8f3870eb9e6425c82b827ecd35888aa]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 22:50:33.967567 19654 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a43f88ce0a244c838c7340b5e779965f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 22:50:33.967612 19916 tablet_service.cc:1467] Processing CreateTablet for tablet a43f88ce0a244c838c7340b5e779965f (DEFAULT_TABLE table=test-table [id=b8f3870eb9e6425c82b827ecd35888aa]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 22:50:33.967940 19916 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a43f88ce0a244c838c7340b5e779965f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 22:50:33.970294 19997 tablet_bootstrap.cc:492] T a43f88ce0a244c838c7340b5e779965f P 11cdc23f46eb4e20bec6d48401a1758c: Bootstrap starting.
I20260929 22:50:33.970361 19785 tablet_service.cc:1467] Processing CreateTablet for tablet a43f88ce0a244c838c7340b5e779965f (DEFAULT_TABLE table=test-table [id=b8f3870eb9e6425c82b827ecd35888aa]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 22:50:33.970649 19785 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a43f88ce0a244c838c7340b5e779965f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 22:50:33.970901 19997 tablet_bootstrap.cc:654] T a43f88ce0a244c838c7340b5e779965f P 11cdc23f46eb4e20bec6d48401a1758c: Neither blocks nor log segments found. Creating new log.
I20260929 22:50:33.971158 19997 log.cc:824] T a43f88ce0a244c838c7340b5e779965f P 11cdc23f46eb4e20bec6d48401a1758c: Log is configured to *not* fsync() on all Append() calls
I20260929 22:50:33.971606 19996 tablet_bootstrap.cc:492] T a43f88ce0a244c838c7340b5e779965f P 184753f0d08d4a23bc3c0fc73380a0e5: Bootstrap starting.
I20260929 22:50:33.971850 19997 tablet_bootstrap.cc:492] T a43f88ce0a244c838c7340b5e779965f P 11cdc23f46eb4e20bec6d48401a1758c: No bootstrap required, opened a new log
I20260929 22:50:33.971905 19997 ts_tablet_manager.cc:1397] T a43f88ce0a244c838c7340b5e779965f P 11cdc23f46eb4e20bec6d48401a1758c: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260929 22:50:33.972237 19996 tablet_bootstrap.cc:654] T a43f88ce0a244c838c7340b5e779965f P 184753f0d08d4a23bc3c0fc73380a0e5: Neither blocks nor log segments found. Creating new log.
I20260929 22:50:33.972538 19996 log.cc:824] T a43f88ce0a244c838c7340b5e779965f P 184753f0d08d4a23bc3c0fc73380a0e5: Log is configured to *not* fsync() on all Append() calls
I20260929 22:50:33.973408 20000 tablet_bootstrap.cc:492] T a43f88ce0a244c838c7340b5e779965f P 7343d1ca698944e09c510ec0227e7322: Bootstrap starting.
I20260929 22:50:33.974044 20000 tablet_bootstrap.cc:654] T a43f88ce0a244c838c7340b5e779965f P 7343d1ca698944e09c510ec0227e7322: Neither blocks nor log segments found. Creating new log.
I20260929 22:50:33.974264 19997 raft_consensus.cc:348] T a43f88ce0a244c838c7340b5e779965f P 11cdc23f46eb4e20bec6d48401a1758c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "11cdc23f46eb4e20bec6d48401a1758c" member_type: VOTER last_known_addr { host: "127.14.147.195" port: 42791 } } peers { permanent_uuid: "184753f0d08d4a23bc3c0fc73380a0e5" member_type: VOTER last_known_addr { host: "127.14.147.193" port: 39067 } } peers { permanent_uuid: "7343d1ca698944e09c510ec0227e7322" member_type: VOTER last_known_addr { host: "127.14.147.194" port: 42977 } }
I20260929 22:50:33.974380 20000 log.cc:824] T a43f88ce0a244c838c7340b5e779965f P 7343d1ca698944e09c510ec0227e7322: Log is configured to *not* fsync() on all Append() calls
I20260929 22:50:33.974433 19997 raft_consensus.cc:374] T a43f88ce0a244c838c7340b5e779965f P 11cdc23f46eb4e20bec6d48401a1758c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 22:50:33.974473 19997 raft_consensus.cc:729] T a43f88ce0a244c838c7340b5e779965f P 11cdc23f46eb4e20bec6d48401a1758c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 11cdc23f46eb4e20bec6d48401a1758c, State: Initialized, Role: FOLLOWER
I20260929 22:50:33.974594 19997 consensus_queue.cc:260] T a43f88ce0a244c838c7340b5e779965f P 11cdc23f46eb4e20bec6d48401a1758c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "11cdc23f46eb4e20bec6d48401a1758c" member_type: VOTER last_known_addr { host: "127.14.147.195" port: 42791 } } peers { permanent_uuid: "184753f0d08d4a23bc3c0fc73380a0e5" member_type: VOTER last_known_addr { host: "127.14.147.193" port: 39067 } } peers { permanent_uuid: "7343d1ca698944e09c510ec0227e7322" member_type: VOTER last_known_addr { host: "127.14.147.194" port: 42977 } }
I20260929 22:50:33.974825 19997 ts_tablet_manager.cc:1428] T a43f88ce0a244c838c7340b5e779965f P 11cdc23f46eb4e20bec6d48401a1758c: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260929 22:50:33.974867 19982 heartbeater.cc:503] Master 127.14.147.254:37211 was elected leader, sending a full tablet report...
I20260929 22:50:33.975301 20000 tablet_bootstrap.cc:492] T a43f88ce0a244c838c7340b5e779965f P 7343d1ca698944e09c510ec0227e7322: No bootstrap required, opened a new log
I20260929 22:50:33.975359 20000 ts_tablet_manager.cc:1397] T a43f88ce0a244c838c7340b5e779965f P 7343d1ca698944e09c510ec0227e7322: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260929 22:50:33.975464 19996 tablet_bootstrap.cc:492] T a43f88ce0a244c838c7340b5e779965f P 184753f0d08d4a23bc3c0fc73380a0e5: No bootstrap required, opened a new log
I20260929 22:50:33.975526 19996 ts_tablet_manager.cc:1397] T a43f88ce0a244c838c7340b5e779965f P 184753f0d08d4a23bc3c0fc73380a0e5: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.000s
I20260929 22:50:33.977511 20000 raft_consensus.cc:348] T a43f88ce0a244c838c7340b5e779965f P 7343d1ca698944e09c510ec0227e7322 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "11cdc23f46eb4e20bec6d48401a1758c" member_type: VOTER last_known_addr { host: "127.14.147.195" port: 42791 } } peers { permanent_uuid: "184753f0d08d4a23bc3c0fc73380a0e5" member_type: VOTER last_known_addr { host: "127.14.147.193" port: 39067 } } peers { permanent_uuid: "7343d1ca698944e09c510ec0227e7322" member_type: VOTER last_known_addr { host: "127.14.147.194" port: 42977 } }
I20260929 22:50:33.977629 20000 raft_consensus.cc:374] T a43f88ce0a244c838c7340b5e779965f P 7343d1ca698944e09c510ec0227e7322 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 22:50:33.977648 20000 raft_consensus.cc:729] T a43f88ce0a244c838c7340b5e779965f P 7343d1ca698944e09c510ec0227e7322 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7343d1ca698944e09c510ec0227e7322, State: Initialized, Role: FOLLOWER
I20260929 22:50:33.977728 20000 consensus_queue.cc:260] T a43f88ce0a244c838c7340b5e779965f P 7343d1ca698944e09c510ec0227e7322 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "11cdc23f46eb4e20bec6d48401a1758c" member_type: VOTER last_known_addr { host: "127.14.147.195" port: 42791 } } peers { permanent_uuid: "184753f0d08d4a23bc3c0fc73380a0e5" member_type: VOTER last_known_addr { host: "127.14.147.193" port: 39067 } } peers { permanent_uuid: "7343d1ca698944e09c510ec0227e7322" member_type: VOTER last_known_addr { host: "127.14.147.194" port: 42977 } }
I20260929 22:50:33.977957 20000 ts_tablet_manager.cc:1428] T a43f88ce0a244c838c7340b5e779965f P 7343d1ca698944e09c510ec0227e7322: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260929 22:50:33.978001 19851 heartbeater.cc:503] Master 127.14.147.254:37211 was elected leader, sending a full tablet report...
I20260929 22:50:33.977926 19996 raft_consensus.cc:348] T a43f88ce0a244c838c7340b5e779965f P 184753f0d08d4a23bc3c0fc73380a0e5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "11cdc23f46eb4e20bec6d48401a1758c" member_type: VOTER last_known_addr { host: "127.14.147.195" port: 42791 } } peers { permanent_uuid: "184753f0d08d4a23bc3c0fc73380a0e5" member_type: VOTER last_known_addr { host: "127.14.147.193" port: 39067 } } peers { permanent_uuid: "7343d1ca698944e09c510ec0227e7322" member_type: VOTER last_known_addr { host: "127.14.147.194" port: 42977 } }
I20260929 22:50:33.978065 19996 raft_consensus.cc:374] T a43f88ce0a244c838c7340b5e779965f P 184753f0d08d4a23bc3c0fc73380a0e5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 22:50:33.978092 19996 raft_consensus.cc:729] T a43f88ce0a244c838c7340b5e779965f P 184753f0d08d4a23bc3c0fc73380a0e5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 184753f0d08d4a23bc3c0fc73380a0e5, State: Initialized, Role: FOLLOWER
I20260929 22:50:33.978195 19996 consensus_queue.cc:260] T a43f88ce0a244c838c7340b5e779965f P 184753f0d08d4a23bc3c0fc73380a0e5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "11cdc23f46eb4e20bec6d48401a1758c" member_type: VOTER last_known_addr { host: "127.14.147.195" port: 42791 } } peers { permanent_uuid: "184753f0d08d4a23bc3c0fc73380a0e5" member_type: VOTER last_known_addr { host: "127.14.147.193" port: 39067 } } peers { permanent_uuid: "7343d1ca698944e09c510ec0227e7322" member_type: VOTER last_known_addr { host: "127.14.147.194" port: 42977 } }
I20260929 22:50:33.978428 19996 ts_tablet_manager.cc:1428] T a43f88ce0a244c838c7340b5e779965f P 184753f0d08d4a23bc3c0fc73380a0e5: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260929 22:50:33.978462 19720 heartbeater.cc:503] Master 127.14.147.254:37211 was elected leader, sending a full tablet report...
I20260929 22:50:34.017725 20004 raft_consensus.cc:482] T a43f88ce0a244c838c7340b5e779965f P 184753f0d08d4a23bc3c0fc73380a0e5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 22:50:34.017861 20004 raft_consensus.cc:504] T a43f88ce0a244c838c7340b5e779965f P 184753f0d08d4a23bc3c0fc73380a0e5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "11cdc23f46eb4e20bec6d48401a1758c" member_type: VOTER last_known_addr { host: "127.14.147.195" port: 42791 } } peers { permanent_uuid: "184753f0d08d4a23bc3c0fc73380a0e5" member_type: VOTER last_known_addr { host: "127.14.147.193" port: 39067 } } peers { permanent_uuid: "7343d1ca698944e09c510ec0227e7322" member_type: VOTER last_known_addr { host: "127.14.147.194" port: 42977 } }
I20260929 22:50:34.018230 20004 leader_election.cc:288] T a43f88ce0a244c838c7340b5e779965f P 184753f0d08d4a23bc3c0fc73380a0e5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 11cdc23f46eb4e20bec6d48401a1758c (127.14.147.195:42791), 7343d1ca698944e09c510ec0227e7322 (127.14.147.194:42977)
I20260929 22:50:34.019511 19993 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:50:34.018545 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.147.193:57447 (local address 127.14.147.194:42977)
0929 22:50:34.018591 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0929 22:50:34.018593 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 22:50:34.018602 (+ 9us) server_negotiation.cc:407] Connection header received
0929 22:50:34.018626 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 22:50:34.018626 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 22:50:34.018630 (+ 4us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 22:50:34.018693 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0929 22:50:34.019254 (+ 561us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 22:50:34.019255 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 22:50:34.019256 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 22:50:34.019261 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0929 22:50:34.019292 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 22:50:34.019296 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 22:50:34.019353 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0929 22:50:34.019406 (+ 53us) server_negotiation.cc:299] Negotiation successful
0929 22:50:34.019440 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260929 22:50:34.019515 19722 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:50:34.018434 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.147.194:42977 (local address 127.14.147.193:57447)
0929 22:50:34.018487 (+ 53us) negotiation.cc:107] Waiting for socket to connect
0929 22:50:34.018497 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0929 22:50:34.018538 (+ 41us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 22:50:34.018722 (+ 184us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 22:50:34.018723 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 22:50:34.018726 (+ 3us) client_negotiation.cc:189] Negotiated authn=SASL
0929 22:50:34.019146 (+ 420us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 22:50:34.019147 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0929 22:50:34.019155 (+ 8us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 22:50:34.019171 (+ 16us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 22:50:34.019356 (+ 185us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 22:50:34.019357 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 22:50:34.019358 (+ 1us) client_negotiation.cc:769] Sending connection context
0929 22:50:34.019405 (+ 47us) client_negotiation.cc:240] Negotiation successful
0929 22:50:34.019442 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":32}
I20260929 22:50:34.019614 19992 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:50:34.018499 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.147.193:46803 (local address 127.14.147.195:42791)
0929 22:50:34.018554 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0929 22:50:34.018556 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 22:50:34.018831 (+ 275us) server_negotiation.cc:407] Connection header received
0929 22:50:34.018865 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 22:50:34.018865 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 22:50:34.018870 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 22:50:34.018918 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0929 22:50:34.019369 (+ 451us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 22:50:34.019370 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 22:50:34.019371 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 22:50:34.019375 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0929 22:50:34.019393 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 22:50:34.019398 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 22:50:34.019466 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0929 22:50:34.019530 (+ 64us) server_negotiation.cc:299] Negotiation successful
0929 22:50:34.019542 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260929 22:50:34.019790 20005 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:50:34.018439 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.147.195:42791 (local address 127.14.147.193:46803)
0929 22:50:34.018756 (+ 317us) negotiation.cc:107] Waiting for socket to connect
0929 22:50:34.018767 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0929 22:50:34.018820 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 22:50:34.018926 (+ 106us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 22:50:34.018927 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 22:50:34.018929 (+ 2us) client_negotiation.cc:189] Negotiated authn=SASL
0929 22:50:34.019275 (+ 346us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 22:50:34.019276 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0929 22:50:34.019282 (+ 6us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 22:50:34.019293 (+ 11us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 22:50:34.019478 (+ 185us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 22:50:34.019479 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 22:50:34.019480 (+ 1us) client_negotiation.cc:769] Sending connection context
0929 22:50:34.019521 (+ 41us) client_negotiation.cc:240] Negotiation successful
0929 22:50:34.019533 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":301,"mutex_wait_us":144,"thread_start_us":105,"threads_started":1}
I20260929 22:50:34.019883 19805 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a43f88ce0a244c838c7340b5e779965f" candidate_uuid: "184753f0d08d4a23bc3c0fc73380a0e5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7343d1ca698944e09c510ec0227e7322" is_pre_election: true
I20260929 22:50:34.020040 19805 raft_consensus.cc:2454] T a43f88ce0a244c838c7340b5e779965f P 7343d1ca698944e09c510ec0227e7322 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 184753f0d08d4a23bc3c0fc73380a0e5 in term 0.
I20260929 22:50:34.020366 19936 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a43f88ce0a244c838c7340b5e779965f" candidate_uuid: "184753f0d08d4a23bc3c0fc73380a0e5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "11cdc23f46eb4e20bec6d48401a1758c" is_pre_election: true
I20260929 22:50:34.020406 19609 leader_election.cc:302] T a43f88ce0a244c838c7340b5e779965f P 184753f0d08d4a23bc3c0fc73380a0e5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 184753f0d08d4a23bc3c0fc73380a0e5, 7343d1ca698944e09c510ec0227e7322; no voters:
I20260929 22:50:34.020493 19936 raft_consensus.cc:2454] T a43f88ce0a244c838c7340b5e779965f P 11cdc23f46eb4e20bec6d48401a1758c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 184753f0d08d4a23bc3c0fc73380a0e5 in term 0.
I20260929 22:50:34.020746 20004 raft_consensus.cc:2781] T a43f88ce0a244c838c7340b5e779965f P 184753f0d08d4a23bc3c0fc73380a0e5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 22:50:34.020807 20004 raft_consensus.cc:482] T a43f88ce0a244c838c7340b5e779965f P 184753f0d08d4a23bc3c0fc73380a0e5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 22:50:34.020829 20004 raft_consensus.cc:3037] T a43f88ce0a244c838c7340b5e779965f P 184753f0d08d4a23bc3c0fc73380a0e5 [term 0 FOLLOWER]: Advancing to term 1
I20260929 22:50:34.021544 20004 raft_consensus.cc:504] T a43f88ce0a244c838c7340b5e779965f P 184753f0d08d4a23bc3c0fc73380a0e5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "11cdc23f46eb4e20bec6d48401a1758c" member_type: VOTER last_known_addr { host: "127.14.147.195" port: 42791 } } peers { permanent_uuid: "184753f0d08d4a23bc3c0fc73380a0e5" member_type: VOTER last_known_addr { host: "127.14.147.193" port: 39067 } } peers { permanent_uuid: "7343d1ca698944e09c510ec0227e7322" member_type: VOTER last_known_addr { host: "127.14.147.194" port: 42977 } }
I20260929 22:50:34.021677 20004 leader_election.cc:288] T a43f88ce0a244c838c7340b5e779965f P 184753f0d08d4a23bc3c0fc73380a0e5 [CANDIDATE]: Term 1 election: Requested vote from peers 11cdc23f46eb4e20bec6d48401a1758c (127.14.147.195:42791), 7343d1ca698944e09c510ec0227e7322 (127.14.147.194:42977)
I20260929 22:50:34.021816 19805 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a43f88ce0a244c838c7340b5e779965f" candidate_uuid: "184753f0d08d4a23bc3c0fc73380a0e5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7343d1ca698944e09c510ec0227e7322"
I20260929 22:50:34.021827 19936 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a43f88ce0a244c838c7340b5e779965f" candidate_uuid: "184753f0d08d4a23bc3c0fc73380a0e5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "11cdc23f46eb4e20bec6d48401a1758c"
I20260929 22:50:34.021890 19805 raft_consensus.cc:3037] T a43f88ce0a244c838c7340b5e779965f P 7343d1ca698944e09c510ec0227e7322 [term 0 FOLLOWER]: Advancing to term 1
I20260929 22:50:34.021898 19936 raft_consensus.cc:3037] T a43f88ce0a244c838c7340b5e779965f P 11cdc23f46eb4e20bec6d48401a1758c [term 0 FOLLOWER]: Advancing to term 1
I20260929 22:50:34.022554 19805 raft_consensus.cc:2454] T a43f88ce0a244c838c7340b5e779965f P 7343d1ca698944e09c510ec0227e7322 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 184753f0d08d4a23bc3c0fc73380a0e5 in term 1.
I20260929 22:50:34.022650 19936 raft_consensus.cc:2454] T a43f88ce0a244c838c7340b5e779965f P 11cdc23f46eb4e20bec6d48401a1758c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 184753f0d08d4a23bc3c0fc73380a0e5 in term 1.
I20260929 22:50:34.022765 19609 leader_election.cc:302] T a43f88ce0a244c838c7340b5e779965f P 184753f0d08d4a23bc3c0fc73380a0e5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 184753f0d08d4a23bc3c0fc73380a0e5, 7343d1ca698944e09c510ec0227e7322; no voters:
I20260929 22:50:34.022917 20004 raft_consensus.cc:2781] T a43f88ce0a244c838c7340b5e779965f P 184753f0d08d4a23bc3c0fc73380a0e5 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 22:50:34.023087 20004 raft_consensus.cc:686] T a43f88ce0a244c838c7340b5e779965f P 184753f0d08d4a23bc3c0fc73380a0e5 [term 1 LEADER]: Becoming Leader. State: Replica: 184753f0d08d4a23bc3c0fc73380a0e5, State: Running, Role: LEADER
I20260929 22:50:34.023195 20004 consensus_queue.cc:237] T a43f88ce0a244c838c7340b5e779965f P 184753f0d08d4a23bc3c0fc73380a0e5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "11cdc23f46eb4e20bec6d48401a1758c" member_type: VOTER last_known_addr { host: "127.14.147.195" port: 42791 } } peers { permanent_uuid: "184753f0d08d4a23bc3c0fc73380a0e5" member_type: VOTER last_known_addr { host: "127.14.147.193" port: 39067 } } peers { permanent_uuid: "7343d1ca698944e09c510ec0227e7322" member_type: VOTER last_known_addr { host: "127.14.147.194" port: 42977 } }
I20260929 22:50:34.024037 19531 catalog_manager.cc:5495] T a43f88ce0a244c838c7340b5e779965f P 184753f0d08d4a23bc3c0fc73380a0e5 reported cstate change: term changed from 0 to 1, leader changed from <none> to 184753f0d08d4a23bc3c0fc73380a0e5 (127.14.147.193). New cstate: current_term: 1 leader_uuid: "184753f0d08d4a23bc3c0fc73380a0e5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "11cdc23f46eb4e20bec6d48401a1758c" member_type: VOTER last_known_addr { host: "127.14.147.195" port: 42791 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "184753f0d08d4a23bc3c0fc73380a0e5" member_type: VOTER last_known_addr { host: "127.14.147.193" port: 39067 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7343d1ca698944e09c510ec0227e7322" member_type: VOTER last_known_addr { host: "127.14.147.194" port: 42977 } health_report { overall_health: UNKNOWN } } }
I20260929 22:50:34.041736 19995 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:50:34.040850 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39932 (local address 127.14.147.193:39067)
0929 22:50:34.040939 (+ 89us) server_negotiation.cc:206] Beginning negotiation
0929 22:50:34.040941 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 22:50:34.040948 (+ 7us) server_negotiation.cc:407] Connection header received
0929 22:50:34.040988 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 22:50:34.040988 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 22:50:34.041003 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 22:50:34.041064 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0929 22:50:34.041538 (+ 474us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 22:50:34.041539 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 22:50:34.041540 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 22:50:34.041544 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0929 22:50:34.041564 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 22:50:34.041569 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 22:50:34.041612 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0929 22:50:34.041644 (+ 32us) server_negotiation.cc:299] Negotiation successful
0929 22:50:34.041656 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260929 22:50:34.043565 19936 raft_consensus.cc:1261] T a43f88ce0a244c838c7340b5e779965f P 11cdc23f46eb4e20bec6d48401a1758c [term 1 FOLLOWER]: Refusing update from remote peer 184753f0d08d4a23bc3c0fc73380a0e5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 22:50:34.043566 19805 raft_consensus.cc:1261] T a43f88ce0a244c838c7340b5e779965f P 7343d1ca698944e09c510ec0227e7322 [term 1 FOLLOWER]: Refusing update from remote peer 184753f0d08d4a23bc3c0fc73380a0e5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 22:50:34.043891 20004 consensus_queue.cc:1035] T a43f88ce0a244c838c7340b5e779965f P 184753f0d08d4a23bc3c0fc73380a0e5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "11cdc23f46eb4e20bec6d48401a1758c" member_type: VOTER last_known_addr { host: "127.14.147.195" port: 42791 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 22:50:34.044036 20006 consensus_queue.cc:1035] T a43f88ce0a244c838c7340b5e779965f P 184753f0d08d4a23bc3c0fc73380a0e5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7343d1ca698944e09c510ec0227e7322" member_type: VOTER last_known_addr { host: "127.14.147.194" port: 42977 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 22:50:34.047114 20010 mvcc.cc:205] Tried to move back new op lower bound from 7334798270639595520 to 7334798270559703040. Current Snapshot: MvccSnapshot[applied={T|T < 7334798270639595520 or (T in {7334798270639595520})}]
I20260929 22:50:34.047122 20011 mvcc.cc:205] Tried to move back new op lower bound from 7334798270639595520 to 7334798270559703040. Current Snapshot: MvccSnapshot[applied={T|T < 7334798270639595520}]
I20260929 22:50:34.049379 19531 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:52866:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 22:50:34.049474 19531 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:52866:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 22:50:34.050220 19531 catalog_manager.cc:5782] T 00000000000000000000000000000000 P dc3a2ea217c7450ea35c47bbd3d14ecb: Sending DeleteTablet for 3 replicas of tablet a43f88ce0a244c838c7340b5e779965f
I20260929 22:50:34.050563 19916 tablet_service.cc:1514] Processing DeleteTablet for tablet a43f88ce0a244c838c7340b5e779965f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 22:50:34 UTC) from {username='slave'} at 127.0.0.1:51362
I20260929 22:50:34.050622 19654 tablet_service.cc:1514] Processing DeleteTablet for tablet a43f88ce0a244c838c7340b5e779965f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 22:50:34 UTC) from {username='slave'} at 127.0.0.1:39922
I20260929 22:50:34.050714 19785 tablet_service.cc:1514] Processing DeleteTablet for tablet a43f88ce0a244c838c7340b5e779965f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 22:50:34 UTC) from {username='slave'} at 127.0.0.1:41822
I20260929 22:50:34.050892 20018 tablet_replica.cc:330] T a43f88ce0a244c838c7340b5e779965f P 11cdc23f46eb4e20bec6d48401a1758c: stopping tablet replica
I20260929 22:50:34.050917 20020 tablet_replica.cc:330] T a43f88ce0a244c838c7340b5e779965f P 7343d1ca698944e09c510ec0227e7322: stopping tablet replica
I20260929 22:50:34.050994 14927 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhWOCLk/build/release/bin/kudu with pid 19592
I20260929 22:50:34.051009 20018 raft_consensus.cc:2229] T a43f88ce0a244c838c7340b5e779965f P 11cdc23f46eb4e20bec6d48401a1758c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 22:50:34.051018 20020 raft_consensus.cc:2229] T a43f88ce0a244c838c7340b5e779965f P 7343d1ca698944e09c510ec0227e7322 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 22:50:34.051115 20018 raft_consensus.cc:2258] T a43f88ce0a244c838c7340b5e779965f P 11cdc23f46eb4e20bec6d48401a1758c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 22:50:34.051121 20020 raft_consensus.cc:2258] T a43f88ce0a244c838c7340b5e779965f P 7343d1ca698944e09c510ec0227e7322 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 22:50:34.051903 20020 ts_tablet_manager.cc:1905] T a43f88ce0a244c838c7340b5e779965f P 7343d1ca698944e09c510ec0227e7322: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 22:50:34.052634 20018 ts_tablet_manager.cc:1905] T a43f88ce0a244c838c7340b5e779965f P 11cdc23f46eb4e20bec6d48401a1758c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 22:50:34.053620 20020 ts_tablet_manager.cc:1918] T a43f88ce0a244c838c7340b5e779965f P 7343d1ca698944e09c510ec0227e7322: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 22:50:34.053705 20020 log.cc:1196] T a43f88ce0a244c838c7340b5e779965f P 7343d1ca698944e09c510ec0227e7322: Deleting WAL directory at /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790722213368738-14927-0/minicluster-data/ts-1/wal/wals/a43f88ce0a244c838c7340b5e779965f
I20260929 22:50:34.053952 20020 ts_tablet_manager.cc:1939] T a43f88ce0a244c838c7340b5e779965f P 7343d1ca698944e09c510ec0227e7322: Deleting consensus metadata
I20260929 22:50:34.054395 20018 ts_tablet_manager.cc:1918] T a43f88ce0a244c838c7340b5e779965f P 11cdc23f46eb4e20bec6d48401a1758c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 22:50:34.054469 20018 log.cc:1196] T a43f88ce0a244c838c7340b5e779965f P 11cdc23f46eb4e20bec6d48401a1758c: Deleting WAL directory at /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790722213368738-14927-0/minicluster-data/ts-2/wal/wals/a43f88ce0a244c838c7340b5e779965f
I20260929 22:50:34.054448 19518 catalog_manager.cc:4841] TS 7343d1ca698944e09c510ec0227e7322 (127.14.147.194:42977): tablet a43f88ce0a244c838c7340b5e779965f (table test-table [id=b8f3870eb9e6425c82b827ecd35888aa]) successfully deleted
I20260929 22:50:34.054695 20018 ts_tablet_manager.cc:1939] T a43f88ce0a244c838c7340b5e779965f P 11cdc23f46eb4e20bec6d48401a1758c: Deleting consensus metadata
I20260929 22:50:34.055081 19519 catalog_manager.cc:4841] TS 11cdc23f46eb4e20bec6d48401a1758c (127.14.147.195:42791): tablet a43f88ce0a244c838c7340b5e779965f (table test-table [id=b8f3870eb9e6425c82b827ecd35888aa]) successfully deleted
W20260929 22:50:34.056761 19519 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.14.147.193:39067 (error 108)
I20260929 22:50:34.056962 14927 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhWOCLk/build/release/bin/kudu with pid 19723
I20260929 22:50:34.057569 19991 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:50:34.057070 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.147.193:39067 (local address 127.0.0.1:39934)
0929 22:50:34.057491 (+ 421us) negotiation.cc:107] Waiting for socket to connect
0929 22:50:34.057534 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.147.193:39067: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":397}
W20260929 22:50:34.057811 19519 catalog_manager.cc:4584] TS 184753f0d08d4a23bc3c0fc73380a0e5 (127.14.147.193:39067): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet a43f88ce0a244c838c7340b5e779965f: Network error: Client connection negotiation failed: client connection to 127.14.147.193:39067: connect: Connection refused (error 111)
I20260929 22:50:34.061681 14927 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhWOCLk/build/release/bin/kudu with pid 19854
I20260929 22:50:34.065614 14927 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhWOCLk/build/release/bin/kudu with pid 19501
2026-09-29T22:50:34Z chronyd exiting
[ OK ] SecurityITest.TestDisableAuthenticationEncryption (651 ms)
[ RUN ] SecurityITest.TestJwtMiniCluster
I20260929 22:50:34.084197 14927 mini_oidc.cc:152] Starting JWKS server
I20260929 22:50:34.085238 14927 webserver.cc:460] Webserver started at https://127.0.0.1:43405/ using document root <none> and password file <none>
I20260929 22:50:34.085327 14927 mini_oidc.cc:189] Starting OIDC Discovery server
I20260929 22:50:34.085484 14927 webserver.cc:460] Webserver started at http://127.0.0.1:33301/ using document root <none> and password file <none>
2026-09-29T22:50:34Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T22:50:34Z Disabled control of system clock
I20260929 22:50:34.093045 14927 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhWOCLk/build/release/bin/kudu
/tmp/dist-test-taskhWOCLk/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790722213368738-14927-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790722213368738-14927-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790722213368738-14927-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790722213368738-14927-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.147.254:43989
--webserver_interface=127.14.147.254
--webserver_port=0
--builtin_ntp_servers=127.14.147.212:34695
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.147.254:43989
--enable_jwt_token_auth=true
--jwks_url=https://localhost:43405/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790722213368738-14927-0/testchainca.cert with env {}
W20260929 22:50:34.173228 20027 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 22:50:34.173448 20027 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 22:50:34.173471 20027 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 22:50:34.174643 20027 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 22:50:34.174683 20027 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 22:50:34.174695 20027 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 22:50:34.174705 20027 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 22:50:34.174715 20027 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260929 22:50:34.174726 20027 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260929 22:50:34.174736 20027 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:43405/jwks.json
I20260929 22:50:34.176388 20027 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.147.212:34695
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790722213368738-14927-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790722213368738-14927-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.14.147.254:43989
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.147.254:43989
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:43405/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790722213368738-14927-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.14.147.254
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790722213368738-14927-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20027
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790722213368738-14927-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 21:58:54 UTC on ccycloud.quasar-xguoti.root.comops.site
build id 4681390
I20260929 22:50:34.176596 20027 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 22:50:34.176772 20027 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 22:50:34.181281 20032 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 22:50:34.181295 20035 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 22:50:34.181286 20033 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 22:50:34.181517 20027 server_base.cc:1034] running on GCE node
I20260929 22:50:34.181756 20027 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 22:50:34.182000 20027 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 22:50:34.183151 20027 hybrid_clock.cc:648] HybridClock initialized: now 1790722234183127 us; error 40 us; skew 500 ppm
I20260929 22:50:34.184392 20027 webserver.cc:460] Webserver started at http://127.14.147.254:40103/ using document root <none> and password file <none>
I20260929 22:50:34.184608 20027 fs_manager.cc:359] Metadata directory not provided
I20260929 22:50:34.184652 20027 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 22:50:34.184792 20027 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 22:50:34.185849 20027 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790722213368738-14927-0/minicluster-data/master-0/data/instance:
uuid: "cb407c3ff51045d291399a6c8737ec95"
format_stamp: "Formatted at 2026-09-29 22:50:34 on dist-test-slave-98mk"
server_key: "50453f20a6bbf06570f12edeeed201e8"
server_key_iv: "d42b74c4e7a7092b91af3d2d06e121b2"
server_key_version: "encryptionkey@0"
I20260929 22:50:34.186167 20027 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790722213368738-14927-0/minicluster-data/master-0/wal/instance:
uuid: "cb407c3ff51045d291399a6c8737ec95"
format_stamp: "Formatted at 2026-09-29 22:50:34 on dist-test-slave-98mk"
server_key: "50453f20a6bbf06570f12edeeed201e8"
server_key_iv: "d42b74c4e7a7092b91af3d2d06e121b2"
server_key_version: "encryptionkey@0"
I20260929 22:50:34.187326 20027 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260929 22:50:34.188104 20041 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 22:50:34.188263 20027 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260929 22:50:34.188342 20027 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790722213368738-14927-0/minicluster-data/master-0/data,/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790722213368738-14927-0/minicluster-data/master-0/wal
uuid: "cb407c3ff51045d291399a6c8737ec95"
format_stamp: "Formatted at 2026-09-29 22:50:34 on dist-test-slave-98mk"
server_key: "50453f20a6bbf06570f12edeeed201e8"
server_key_iv: "d42b74c4e7a7092b91af3d2d06e121b2"
server_key_version: "encryptionkey@0"
I20260929 22:50:34.188400 20027 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790722213368738-14927-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790722213368738-14927-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790722213368738-14927-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 22:50:34.200947 20027 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 22:50:34.201263 20027 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 22:50:34.201395 20027 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 22:50:34.205191 20027 rpc_server.cc:307] RPC server started. Bound to: 127.14.147.254:43989
I20260929 22:50:34.205229 20093 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.147.254:43989 every 8 connection(s)
I20260929 22:50:34.205549 20027 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790722213368738-14927-0/minicluster-data/master-0/data/info.pb
I20260929 22:50:34.206440 20094 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 22:50:34.207530 14927 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhWOCLk/build/release/bin/kudu as pid 20027
I20260929 22:50:34.207643 14927 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790722213368738-14927-0/minicluster-data/master-0/wal/instance
I20260929 22:50:34.207815 14927 external_mini_cluster.cc:1428] Setting key 7a6f150a8c91da4f5adb04f4c4f82bc2
I20260929 22:50:34.209544 20094 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cb407c3ff51045d291399a6c8737ec95: Bootstrap starting.
I20260929 22:50:34.210319 20094 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P cb407c3ff51045d291399a6c8737ec95: Neither blocks nor log segments found. Creating new log.
I20260929 22:50:34.210605 20094 log.cc:824] T 00000000000000000000000000000000 P cb407c3ff51045d291399a6c8737ec95: Log is configured to *not* fsync() on all Append() calls
I20260929 22:50:34.211364 20094 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cb407c3ff51045d291399a6c8737ec95: No bootstrap required, opened a new log
I20260929 22:50:34.214154 20094 raft_consensus.cc:348] T 00000000000000000000000000000000 P cb407c3ff51045d291399a6c8737ec95 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cb407c3ff51045d291399a6c8737ec95" member_type: VOTER last_known_addr { host: "127.14.147.254" port: 43989 } }
I20260929 22:50:34.214344 20094 raft_consensus.cc:374] T 00000000000000000000000000000000 P cb407c3ff51045d291399a6c8737ec95 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 22:50:34.214380 20097 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:50:34.208398 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56800 (local address 127.14.147.254:43989)
0929 22:50:34.208607 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0929 22:50:34.208615 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 22:50:34.208650 (+ 35us) server_negotiation.cc:407] Connection header received
0929 22:50:34.209264 (+ 614us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 22:50:34.209273 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 22:50:34.209365 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 22:50:34.209521 (+ 156us) server_negotiation.cc:226] Negotiated authn=SASL
0929 22:50:34.209761 (+ 240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:50:34.211049 (+ 1288us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:50:34.211907 (+ 858us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:50:34.212804 (+ 897us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:50:34.212962 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 22:50:34.213592 (+ 630us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 22:50:34.213598 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 22:50:34.213599 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 22:50:34.213831 (+ 232us) server_negotiation.cc:922] calling sasl_server_start()
0929 22:50:34.213876 (+ 45us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 22:50:34.213891 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 22:50:34.213960 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0929 22:50:34.214078 (+ 118us) server_negotiation.cc:299] Negotiation successful
0929 22:50:34.214169 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":132,"thread_start_us":82,"threads_started":1}
I20260929 22:50:34.214378 20094 raft_consensus.cc:729] T 00000000000000000000000000000000 P cb407c3ff51045d291399a6c8737ec95 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cb407c3ff51045d291399a6c8737ec95, State: Initialized, Role: FOLLOWER
I20260929 22:50:34.214532 20094 consensus_queue.cc:260] T 00000000000000000000000000000000 P cb407c3ff51045d291399a6c8737ec95 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cb407c3ff51045d291399a6c8737ec95" member_type: VOTER last_known_addr { host: "127.14.147.254" port: 43989 } }
I20260929 22:50:34.214612 20094 raft_consensus.cc:388] T 00000000000000000000000000000000 P cb407c3ff51045d291399a6c8737ec95 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 22:50:34.214645 20094 raft_consensus.cc:482] T 00000000000000000000000000000000 P cb407c3ff51045d291399a6c8737ec95 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 22:50:34.214684 20094 raft_consensus.cc:3037] T 00000000000000000000000000000000 P cb407c3ff51045d291399a6c8737ec95 [term 0 FOLLOWER]: Advancing to term 1
I20260929 22:50:34.215312 14927 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 22:50:34.215374 20094 raft_consensus.cc:504] T 00000000000000000000000000000000 P cb407c3ff51045d291399a6c8737ec95 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cb407c3ff51045d291399a6c8737ec95" member_type: VOTER last_known_addr { host: "127.14.147.254" port: 43989 } }
I20260929 22:50:34.215514 20094 leader_election.cc:302] T 00000000000000000000000000000000 P cb407c3ff51045d291399a6c8737ec95 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: cb407c3ff51045d291399a6c8737ec95; no voters:
I20260929 22:50:34.215696 20094 leader_election.cc:288] T 00000000000000000000000000000000 P cb407c3ff51045d291399a6c8737ec95 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 22:50:34.215739 20099 raft_consensus.cc:2781] T 00000000000000000000000000000000 P cb407c3ff51045d291399a6c8737ec95 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 22:50:34.215860 20099 raft_consensus.cc:686] T 00000000000000000000000000000000 P cb407c3ff51045d291399a6c8737ec95 [term 1 LEADER]: Becoming Leader. State: Replica: cb407c3ff51045d291399a6c8737ec95, State: Running, Role: LEADER
I20260929 22:50:34.215942 20099 consensus_queue.cc:237] T 00000000000000000000000000000000 P cb407c3ff51045d291399a6c8737ec95 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cb407c3ff51045d291399a6c8737ec95" member_type: VOTER last_known_addr { host: "127.14.147.254" port: 43989 } }
I20260929 22:50:34.215976 20094 sys_catalog.cc:564] T 00000000000000000000000000000000 P cb407c3ff51045d291399a6c8737ec95 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 22:50:34.216256 20099 sys_catalog.cc:455] T 00000000000000000000000000000000 P cb407c3ff51045d291399a6c8737ec95 [sys.catalog]: SysCatalogTable state changed. Reason: New leader cb407c3ff51045d291399a6c8737ec95. Latest consensus state: current_term: 1 leader_uuid: "cb407c3ff51045d291399a6c8737ec95" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cb407c3ff51045d291399a6c8737ec95" member_type: VOTER last_known_addr { host: "127.14.147.254" port: 43989 } } }
I20260929 22:50:34.216336 20099 sys_catalog.cc:458] T 00000000000000000000000000000000 P cb407c3ff51045d291399a6c8737ec95 [sys.catalog]: This master's current role is: LEADER
I20260929 22:50:34.216529 20101 sys_catalog.cc:455] T 00000000000000000000000000000000 P cb407c3ff51045d291399a6c8737ec95 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "cb407c3ff51045d291399a6c8737ec95" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cb407c3ff51045d291399a6c8737ec95" member_type: VOTER last_known_addr { host: "127.14.147.254" port: 43989 } } }
I20260929 22:50:34.216616 20101 sys_catalog.cc:458] T 00000000000000000000000000000000 P cb407c3ff51045d291399a6c8737ec95 [sys.catalog]: This master's current role is: LEADER
W20260929 22:50:34.217682 20115 catalog_manager.cc:1558] T 00000000000000000000000000000000 P cb407c3ff51045d291399a6c8737ec95: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 22:50:34.217725 20115 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260929 22:50:34.217780 20103 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 22:50:34.217931 20103 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 22:50:34.219316 20103 catalog_manager.cc:1347] Generated new cluster ID: bc05d3ebd3114938930ae5b441cc1f9b
I20260929 22:50:34.219379 20103 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 22:50:34.238873 20103 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 22:50:34.239655 20103 catalog_manager.cc:1504] Loading token signing keys...
I20260929 22:50:34.244316 20103 catalog_manager.cc:5868] T 00000000000000000000000000000000 P cb407c3ff51045d291399a6c8737ec95: Generated new TSK 0
I20260929 22:50:34.244508 20103 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260929 22:50:34.270560 20124 webserver.cc:533] Webserver: sslize failed: error:14094415:SSL routines:ssl3_read_bytes:sslv3 alert certificate expired
E20260929 22:50:34.270743 20124 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260929 22:50:34.270928 20097 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:50:34.254708 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56810 (local address 127.14.147.254:43989)
0929 22:50:34.254757 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0929 22:50:34.254759 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 22:50:34.254845 (+ 86us) server_negotiation.cc:407] Connection header received
0929 22:50:34.255074 (+ 229us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 22:50:34.255076 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 22:50:34.255089 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 22:50:34.255173 (+ 84us) server_negotiation.cc:226] Negotiated authn=JWT
0929 22:50:34.255301 (+ 128us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:50:34.256098 (+ 797us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:50:34.256855 (+ 757us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:50:34.257155 (+ 300us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:50:34.257233 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 22:50:34.257426 (+ 193us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0929 22:50:34.270701 (+ 13275us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:43405/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0929 22:50:34.270852 (+ 151us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:56810: Failed to load JWKS: Error downloading JWKS from 'https://localhost:43405/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"server-negotiator.queue_time_us":26}
W20260929 22:50:34.270989 20097 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:56810: Failed to load JWKS: Error downloading JWKS from 'https://localhost:43405/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
W20260929 22:50:34.271351 20123 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 22:50:34.254615 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.147.254:43989 (local address 127.0.0.1:56810)
0929 22:50:34.254775 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0929 22:50:34.254786 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0929 22:50:34.254843 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 22:50:34.255177 (+ 334us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 22:50:34.255178 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 22:50:34.255181 (+ 3us) client_negotiation.cc:189] Negotiated authn=JWT
0929 22:50:34.255239 (+ 58us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 22:50:34.255241 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 22:50:34.256173 (+ 932us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 22:50:34.256173 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 22:50:34.256778 (+ 605us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 22:50:34.256780 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 22:50:34.257244 (+ 464us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 22:50:34.257246 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 22:50:34.257297 (+ 51us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 22:50:34.257321 (+ 24us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0929 22:50:34.270935 (+ 13614us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:43405/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0929 22:50:34.271052 (+ 117us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.14.147.254:43989: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:43405/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":75,"threads_started":1}
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:597: Failure
Failed
Bad status: Runtime error: Could not connect to the cluster: Client connection negotiation failed: client connection to 127.14.147.254:43989: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:43405/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Google Test trace:
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:593: Valid JWT
I20260929 22:50:34.271781 14927 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhWOCLk/build/release/bin/kudu with pid 20027
2026-09-29T22:50:34Z chronyd exiting
I20260929 22:50:34.307243 14927 test_util.cc:175] -----------------------------------------------
I20260929 22:50:34.307327 14927 test_util.cc:176] Had fatal failures, leaving test files at /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790722213368738-14927-0
[ FAILED ] SecurityITest.TestJwtMiniCluster (224 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithInvalidCert
I20260929 22:50:34.308156 14927 mini_oidc.cc:152] Starting JWKS server
I20260929 22:50:34.308832 14927 webserver.cc:460] Webserver started at https://127.0.0.1:41585/ using document root <none> and password file <none>
I20260929 22:50:34.308887 14927 mini_oidc.cc:189] Starting OIDC Discovery server
I20260929 22:50:34.309019 14927 webserver.cc:460] Webserver started at http://127.0.0.1:40889/ using document root <none> and password file <none>
2026-09-29T22:50:34Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T22:50:34Z Disabled control of system clock
I20260929 22:50:34.315747 14927 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhWOCLk/build/release/bin/kudu
/tmp/dist-test-taskhWOCLk/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790722213368738-14927-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790722213368738-14927-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790722213368738-14927-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790722213368738-14927-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.147.254:38429
--webserver_interface=127.14.147.254
--webserver_port=0
--builtin_ntp_servers=127.14.147.212:34785
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.147.254:38429
--enable_jwt_token_auth=true
--jwks_url=https://localhost:41585/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790722213368738-14927-0/testchainca.cert with env {}
W20260929 22:50:34.397022 20130 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 22:50:34.397239 20130 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 22:50:34.397262 20130 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 22:50:34.398442 20130 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 22:50:34.398480 20130 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 22:50:34.398491 20130 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 22:50:34.398502 20130 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 22:50:34.398514 20130 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260929 22:50:34.398523 20130 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260929 22:50:34.398533 20130 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:41585/jwks.json
I20260929 22:50:34.400072 20130 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.147.212:34785
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790722213368738-14927-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790722213368738-14927-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.14.147.254:38429
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.147.254:38429
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:41585/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790722213368738-14927-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.14.147.254
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790722213368738-14927-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20130
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790722213368738-14927-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 21:58:54 UTC on ccycloud.quasar-xguoti.root.comops.site
build id 4681390
I20260929 22:50:34.400276 20130 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 22:50:34.400444 20130 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 22:50:34.405099 20136 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 22:50:34.405112 20135 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 22:50:34.405179 20130 server_base.cc:1034] running on GCE node
W20260929 22:50:34.405112 20138 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 22:50:34.405505 20130 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 22:50:34.405777 20130 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 22:50:34.407034 20130 hybrid_clock.cc:648] HybridClock initialized: now 1790722234407004 us; error 29 us; skew 500 ppm
I20260929 22:50:34.408751 20130 webserver.cc:460] Webserver started at http://127.14.147.254:33197/ using document root <none> and password file <none>
I20260929 22:50:34.408993 20130 fs_manager.cc:359] Metadata directory not provided
I20260929 22:50:34.409044 20130 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 22:50:34.409171 20130 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 22:50:34.410296 20130 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790722213368738-14927-0/minicluster-data/master-0/data/instance:
uuid: "7d3bb119207d4cfa88b63e8c257e69c8"
format_stamp: "Formatted at 2026-09-29 22:50:34 on dist-test-slave-98mk"
server_key: "eb66b2caaf94d4b5f92d017b88f78f2a"
server_key_iv: "5f78225c5e7334289681e962fded6be6"
server_key_version: "encryptionkey@0"
I20260929 22:50:34.410619 20130 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790722213368738-14927-0/minicluster-data/master-0/wal/instance:
uuid: "7d3bb119207d4cfa88b63e8c257e69c8"
format_stamp: "Formatted at 2026-09-29 22:50:34 on dist-test-slave-98mk"
server_key: "eb66b2caaf94d4b5f92d017b88f78f2a"
server_key_iv: "5f78225c5e7334289681e962fded6be6"
server_key_version: "encryptionkey@0"
I20260929 22:50:34.411876 20130 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.002s
I20260929 22:50:34.412575 20144 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 22:50:34.412720 20130 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260929 22:50:34.412803 20130 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790722213368738-14927-0/minicluster-data/master-0/data,/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790722213368738-14927-0/minicluster-data/master-0/wal
uuid: "7d3bb119207d4cfa88b63e8c257e69c8"
format_stamp: "Formatted at 2026-09-29 22:50:34 on dist-test-slave-98mk"
server_key: "eb66b2caaf94d4b5f92d017b88f78f2a"
server_key_iv: "5f78225c5e7334289681e962fded6be6"
server_key_version: "encryptionkey@0"
I20260929 22:50:34.412866 20130 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790722213368738-14927-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790722213368738-14927-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790722213368738-14927-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 22:50:34.431906 20130 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 22:50:34.432235 20130 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 22:50:34.432366 20130 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 22:50:34.436084 20130 rpc_server.cc:307] RPC server started. Bound to: 127.14.147.254:38429
I20260929 22:50:34.436122 20197 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.147.254:38429 every 8 connection(s)
I20260929 22:50:34.436437 20130 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790722213368738-14927-0/minicluster-data/master-0/data/info.pb
I20260929 22:50:34.437055 20198 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 22:50:34.439845 20198 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7d3bb119207d4cfa88b63e8c257e69c8: Bootstrap starting.
I20260929 22:50:34.440409 20198 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7d3bb119207d4cfa88b63e8c257e69c8: Neither blocks nor log segments found. Creating new log.
I20260929 22:50:34.440687 20198 log.cc:824] T 00000000000000000000000000000000 P 7d3bb119207d4cfa88b63e8c257e69c8: Log is configured to *not* fsync() on all Append() calls
I20260929 22:50:34.440821 14927 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhWOCLk/build/release/bin/kudu as pid 20130
I20260929 22:50:34.440917 14927 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790722213368738-14927-0/minicluster-data/master-0/wal/instance
I20260929 22:50:34.441109 14927 external_mini_cluster.cc:1428] Setting key c14c98e085befe9fd3072b51a2dda500
I20260929 22:50:34.441354 20198 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7d3bb119207d4cfa88b63e8c257e69c8: No bootstrap required, opened a new log
I20260929 22:50:34.443718 20198 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7d3bb119207d4cfa88b63e8c257e69c8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7d3bb119207d4cfa88b63e8c257e69c8" member_type: VOTER last_known_addr { host: "127.14.147.254" port: 38429 } }
I20260929 22:50:34.443857 20198 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7d3bb119207d4cfa88b63e8c257e69c8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 22:50:34.443890 20198 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7d3bb119207d4cfa88b63e8c257e69c8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7d3bb119207d4cfa88b63e8c257e69c8, State: Initialized, Role: FOLLOWER
I20260929 22:50:34.444023 20198 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7d3bb119207d4cfa88b63e8c257e69c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7d3bb119207d4cfa88b63e8c257e69c8" member_type: VOTER last_known_addr { host: "127.14.147.254" port: 38429 } }
I20260929 22:50:34.444103 20198 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7d3bb119207d4cfa88b63e8c257e69c8 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 22:50:34.444137 20198 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7d3bb119207d4cfa88b63e8c257e69c8 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 22:50:34.444180 20198 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7d3bb119207d4cfa88b63e8c257e69c8 [term 0 FOLLOWER]: Advancing to term 1
I20260929 22:50:34.444875 20198 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7d3bb119207d4cfa88b63e8c257e69c8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7d3bb119207d4cfa88b63e8c257e69c8" member_type: VOTER last_known_addr { host: "127.14.147.254" port: 38429 } }
I20260929 22:50:34.444968 20198 leader_election.cc:302] T 00000000000000000000000000000000 P 7d3bb119207d4cfa88b63e8c257e69c8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7d3bb119207d4cfa88b63e8c257e69c8; no voters:
I20260929 22:50:34.445118 20198 leader_election.cc:288] T 00000000000000000000000000000000 P 7d3bb119207d4cfa88b63e8c257e69c8 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 22:50:34.445152 20203 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7d3bb119207d4cfa88b63e8c257e69c8 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 22:50:34.445257 20203 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7d3bb119207d4cfa88b63e8c257e69c8 [term 1 LEADER]: Becoming Leader. State: Replica: 7d3bb119207d4cfa88b63e8c257e69c8, State: Running, Role: LEADER
I20260929 22:50:34.445339 20203 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7d3bb119207d4cfa88b63e8c257e69c8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7d3bb119207d4cfa88b63e8c257e69c8" member_type: VOTER last_known_addr { host: "127.14.147.254" port: 38429 } }
I20260929 22:50:34.445389 20198 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7d3bb119207d4cfa88b63e8c257e69c8 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 22:50:34.445715 20203 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7d3bb119207d4cfa88b63e8c257e69c8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7d3bb119207d4cfa88b63e8c257e69c8. Latest consensus state: current_term: 1 leader_uuid: "7d3bb119207d4cfa88b63e8c257e69c8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7d3bb119207d4cfa88b63e8c257e69c8" member_type: VOTER last_known_addr { host: "127.14.147.254" port: 38429 } } }
I20260929 22:50:34.445796 20203 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7d3bb119207d4cfa88b63e8c257e69c8 [sys.catalog]: This master's current role is: LEADER
I20260929 22:50:34.445941 20204 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7d3bb119207d4cfa88b63e8c257e69c8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7d3bb119207d4cfa88b63e8c257e69c8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7d3bb119207d4cfa88b63e8c257e69c8" member_type: VOTER last_known_addr { host: "127.14.147.254" port: 38429 } } }
I20260929 22:50:34.446002 20204 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7d3bb119207d4cfa88b63e8c257e69c8 [sys.catalog]: This master's current role is: LEADER
I20260929 22:50:34.446107 20208 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 22:50:34.446616 20208 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 22:50:34.447911 20202 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:50:34.441564 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50946 (local address 127.14.147.254:38429)
0929 22:50:34.441755 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0929 22:50:34.441763 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 22:50:34.441787 (+ 24us) server_negotiation.cc:407] Connection header received
0929 22:50:34.442521 (+ 734us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 22:50:34.442527 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 22:50:34.442585 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 22:50:34.442692 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0929 22:50:34.442912 (+ 220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:50:34.443803 (+ 891us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:50:34.444810 (+ 1007us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:50:34.445331 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:50:34.445767 (+ 436us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 22:50:34.447256 (+ 1489us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 22:50:34.447260 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 22:50:34.447260 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 22:50:34.447454 (+ 194us) server_negotiation.cc:922] calling sasl_server_start()
0929 22:50:34.447501 (+ 47us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 22:50:34.447517 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 22:50:34.447580 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0929 22:50:34.447657 (+ 77us) server_negotiation.cc:299] Negotiation successful
0929 22:50:34.447738 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":116,"thread_start_us":73,"threads_started":1}
I20260929 22:50:34.448454 20208 catalog_manager.cc:1347] Generated new cluster ID: 4c2b9b6c896940ed849e2cc5f043cec9
I20260929 22:50:34.448501 20208 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 22:50:34.448807 14927 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 22:50:34.456419 20208 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 22:50:34.456918 20208 catalog_manager.cc:1504] Loading token signing keys...
I20260929 22:50:34.475162 20208 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7d3bb119207d4cfa88b63e8c257e69c8: Generated new TSK 0
I20260929 22:50:34.475371 20208 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260929 22:50:34.498934 20228 webserver.cc:533] Webserver: sslize failed: error:14094418:SSL routines:ssl3_read_bytes:tlsv1 alert unknown ca
E20260929 22:50:34.499010 20228 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260929 22:50:34.499372 20202 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:50:34.485220 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50952 (local address 127.14.147.254:38429)
0929 22:50:34.485279 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0929 22:50:34.485281 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 22:50:34.485352 (+ 71us) server_negotiation.cc:407] Connection header received
0929 22:50:34.485394 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 22:50:34.485395 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 22:50:34.485408 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 22:50:34.485484 (+ 76us) server_negotiation.cc:226] Negotiated authn=JWT
0929 22:50:34.485591 (+ 107us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:50:34.486378 (+ 787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:50:34.487124 (+ 746us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:50:34.487472 (+ 348us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:50:34.487545 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 22:50:34.487675 (+ 130us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0929 22:50:34.499140 (+ 11465us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:41585/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0929 22:50:34.499298 (+ 158us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:50952: Failed to load JWKS: Error downloading JWKS from 'https://localhost:41585/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"server-negotiator.queue_time_us":34}
W20260929 22:50:34.499459 20202 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:50952: Failed to load JWKS: Error downloading JWKS from 'https://localhost:41585/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
W20260929 22:50:34.499457 20227 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 22:50:34.485150 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.147.254:38429 (local address 127.0.0.1:50952)
0929 22:50:34.485280 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0929 22:50:34.485290 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0929 22:50:34.485349 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 22:50:34.485488 (+ 139us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 22:50:34.485489 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 22:50:34.485490 (+ 1us) client_negotiation.cc:189] Negotiated authn=JWT
0929 22:50:34.485536 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 22:50:34.485538 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 22:50:34.486466 (+ 928us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 22:50:34.486467 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 22:50:34.487052 (+ 585us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 22:50:34.487053 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 22:50:34.487541 (+ 488us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 22:50:34.487541 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 22:50:34.487583 (+ 42us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 22:50:34.487587 (+ 4us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0929 22:50:34.499258 (+ 11671us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:41585/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0929 22:50:34.499359 (+ 101us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.14.147.254:38429: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:41585/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"client-negotiator.queue_time_us":106,"thread_start_us":77,"threads_started":1}
I20260929 22:50:34.499733 14927 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhWOCLk/build/release/bin/kudu with pid 20130
2026-09-29T22:50:34Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithInvalidCert (228 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithUntrustedCert
I20260929 22:50:34.536657 14927 mini_oidc.cc:152] Starting JWKS server
I20260929 22:50:34.537523 14927 webserver.cc:460] Webserver started at https://127.0.0.1:38061/ using document root <none> and password file <none>
I20260929 22:50:34.537585 14927 mini_oidc.cc:189] Starting OIDC Discovery server
I20260929 22:50:34.537730 14927 webserver.cc:460] Webserver started at http://127.0.0.1:41325/ using document root <none> and password file <none>
2026-09-29T22:50:34Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T22:50:34Z Disabled control of system clock
I20260929 22:50:34.544791 14927 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhWOCLk/build/release/bin/kudu
/tmp/dist-test-taskhWOCLk/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790722213368738-14927-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790722213368738-14927-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790722213368738-14927-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790722213368738-14927-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.147.254:38225
--webserver_interface=127.14.147.254
--webserver_port=0
--builtin_ntp_servers=127.14.147.212:37085
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.147.254:38225
--enable_jwt_token_auth=true
--jwks_url=https://localhost:38061/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260929 22:50:34.626513 20234 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 22:50:34.626741 20234 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 22:50:34.626772 20234 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 22:50:34.628072 20234 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 22:50:34.628122 20234 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 22:50:34.628134 20234 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 22:50:34.628144 20234 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 22:50:34.628155 20234 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260929 22:50:34.628166 20234 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260929 22:50:34.628180 20234 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:38061/jwks.json
I20260929 22:50:34.629622 20234 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.147.212:37085
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790722213368738-14927-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790722213368738-14927-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.14.147.254:38225
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.147.254:38225
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:38061/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790722213368738-14927-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.14.147.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20234
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790722213368738-14927-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 21:58:54 UTC on ccycloud.quasar-xguoti.root.comops.site
build id 4681390
I20260929 22:50:34.629889 20234 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 22:50:34.630203 20234 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 22:50:34.634831 20240 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 22:50:34.634824 20239 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 22:50:34.634857 20242 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 22:50:34.635190 20234 server_base.cc:1034] running on GCE node
I20260929 22:50:34.635439 20234 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 22:50:34.635756 20234 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 22:50:34.636894 20234 hybrid_clock.cc:648] HybridClock initialized: now 1790722234636881 us; error 29 us; skew 500 ppm
I20260929 22:50:34.638134 20234 webserver.cc:460] Webserver started at http://127.14.147.254:39535/ using document root <none> and password file <none>
I20260929 22:50:34.638367 20234 fs_manager.cc:359] Metadata directory not provided
I20260929 22:50:34.638435 20234 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 22:50:34.638566 20234 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 22:50:34.639740 20234 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790722213368738-14927-0/minicluster-data/master-0/data/instance:
uuid: "4248c11a8feb41478da5498baf50b3ed"
format_stamp: "Formatted at 2026-09-29 22:50:34 on dist-test-slave-98mk"
server_key: "8dda9ea2624572a0c10d4f9a13a01288"
server_key_iv: "bb680651a18208d7ce6e2df0ba3345ac"
server_key_version: "encryptionkey@0"
I20260929 22:50:34.640067 20234 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790722213368738-14927-0/minicluster-data/master-0/wal/instance:
uuid: "4248c11a8feb41478da5498baf50b3ed"
format_stamp: "Formatted at 2026-09-29 22:50:34 on dist-test-slave-98mk"
server_key: "8dda9ea2624572a0c10d4f9a13a01288"
server_key_iv: "bb680651a18208d7ce6e2df0ba3345ac"
server_key_version: "encryptionkey@0"
I20260929 22:50:34.641399 20234 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260929 22:50:34.642138 20248 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 22:50:34.642295 20234 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260929 22:50:34.642450 20234 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790722213368738-14927-0/minicluster-data/master-0/data,/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790722213368738-14927-0/minicluster-data/master-0/wal
uuid: "4248c11a8feb41478da5498baf50b3ed"
format_stamp: "Formatted at 2026-09-29 22:50:34 on dist-test-slave-98mk"
server_key: "8dda9ea2624572a0c10d4f9a13a01288"
server_key_iv: "bb680651a18208d7ce6e2df0ba3345ac"
server_key_version: "encryptionkey@0"
I20260929 22:50:34.642529 20234 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790722213368738-14927-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790722213368738-14927-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790722213368738-14927-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 22:50:34.664508 20234 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 22:50:34.664849 20234 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 22:50:34.664983 20234 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 22:50:34.668988 20234 rpc_server.cc:307] RPC server started. Bound to: 127.14.147.254:38225
I20260929 22:50:34.669011 20300 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.147.254:38225 every 8 connection(s)
I20260929 22:50:34.669401 20234 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790722213368738-14927-0/minicluster-data/master-0/data/info.pb
I20260929 22:50:34.670029 20301 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 22:50:34.672839 20301 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4248c11a8feb41478da5498baf50b3ed: Bootstrap starting.
I20260929 22:50:34.673431 20301 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4248c11a8feb41478da5498baf50b3ed: Neither blocks nor log segments found. Creating new log.
I20260929 22:50:34.673686 20301 log.cc:824] T 00000000000000000000000000000000 P 4248c11a8feb41478da5498baf50b3ed: Log is configured to *not* fsync() on all Append() calls
I20260929 22:50:34.674350 20301 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4248c11a8feb41478da5498baf50b3ed: No bootstrap required, opened a new log
I20260929 22:50:34.676015 20301 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4248c11a8feb41478da5498baf50b3ed [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4248c11a8feb41478da5498baf50b3ed" member_type: VOTER last_known_addr { host: "127.14.147.254" port: 38225 } }
I20260929 22:50:34.676165 20301 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4248c11a8feb41478da5498baf50b3ed [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 22:50:34.676187 20301 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4248c11a8feb41478da5498baf50b3ed [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4248c11a8feb41478da5498baf50b3ed, State: Initialized, Role: FOLLOWER
I20260929 22:50:34.676313 20301 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4248c11a8feb41478da5498baf50b3ed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4248c11a8feb41478da5498baf50b3ed" member_type: VOTER last_known_addr { host: "127.14.147.254" port: 38225 } }
I20260929 22:50:34.676381 20301 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4248c11a8feb41478da5498baf50b3ed [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 22:50:34.676405 20301 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4248c11a8feb41478da5498baf50b3ed [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 22:50:34.676453 20301 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4248c11a8feb41478da5498baf50b3ed [term 0 FOLLOWER]: Advancing to term 1
I20260929 22:50:34.676985 20301 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4248c11a8feb41478da5498baf50b3ed [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4248c11a8feb41478da5498baf50b3ed" member_type: VOTER last_known_addr { host: "127.14.147.254" port: 38225 } }
I20260929 22:50:34.677078 20301 leader_election.cc:302] T 00000000000000000000000000000000 P 4248c11a8feb41478da5498baf50b3ed [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4248c11a8feb41478da5498baf50b3ed; no voters:
I20260929 22:50:34.677237 20301 leader_election.cc:288] T 00000000000000000000000000000000 P 4248c11a8feb41478da5498baf50b3ed [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 22:50:34.677309 20304 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4248c11a8feb41478da5498baf50b3ed [term 1 FOLLOWER]: Leader election won for term 1
I20260929 22:50:34.677453 20304 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4248c11a8feb41478da5498baf50b3ed [term 1 LEADER]: Becoming Leader. State: Replica: 4248c11a8feb41478da5498baf50b3ed, State: Running, Role: LEADER
I20260929 22:50:34.677481 20301 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4248c11a8feb41478da5498baf50b3ed [sys.catalog]: configured and running, proceeding with master startup.
I20260929 22:50:34.677560 20304 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4248c11a8feb41478da5498baf50b3ed [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4248c11a8feb41478da5498baf50b3ed" member_type: VOTER last_known_addr { host: "127.14.147.254" port: 38225 } }
I20260929 22:50:34.677915 20306 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4248c11a8feb41478da5498baf50b3ed [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4248c11a8feb41478da5498baf50b3ed. Latest consensus state: current_term: 1 leader_uuid: "4248c11a8feb41478da5498baf50b3ed" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4248c11a8feb41478da5498baf50b3ed" member_type: VOTER last_known_addr { host: "127.14.147.254" port: 38225 } } }
I20260929 22:50:34.678030 20306 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4248c11a8feb41478da5498baf50b3ed [sys.catalog]: This master's current role is: LEADER
I20260929 22:50:34.678103 20305 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4248c11a8feb41478da5498baf50b3ed [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4248c11a8feb41478da5498baf50b3ed" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4248c11a8feb41478da5498baf50b3ed" member_type: VOTER last_known_addr { host: "127.14.147.254" port: 38225 } } }
I20260929 22:50:34.678161 20305 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4248c11a8feb41478da5498baf50b3ed [sys.catalog]: This master's current role is: LEADER
I20260929 22:50:34.678287 20313 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 22:50:34.678741 20313 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 22:50:34.679231 14927 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhWOCLk/build/release/bin/kudu as pid 20234
I20260929 22:50:34.679327 14927 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790722213368738-14927-0/minicluster-data/master-0/wal/instance
I20260929 22:50:34.679529 14927 external_mini_cluster.cc:1428] Setting key a7f0b488486f588aeb2765b0398a38a2
I20260929 22:50:34.680377 20313 catalog_manager.cc:1347] Generated new cluster ID: db12c4747a7d4b90835afabb18acff0d
I20260929 22:50:34.680437 20313 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 22:50:34.685868 20324 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:50:34.679968 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33710 (local address 127.14.147.254:38225)
0929 22:50:34.680147 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0929 22:50:34.680156 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 22:50:34.680545 (+ 389us) server_negotiation.cc:407] Connection header received
0929 22:50:34.680702 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 22:50:34.680710 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 22:50:34.680797 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 22:50:34.680958 (+ 161us) server_negotiation.cc:226] Negotiated authn=SASL
0929 22:50:34.681151 (+ 193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:50:34.682569 (+ 1418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:50:34.683684 (+ 1115us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:50:34.684349 (+ 665us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:50:34.684486 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 22:50:34.685133 (+ 647us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 22:50:34.685138 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 22:50:34.685138 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 22:50:34.685362 (+ 224us) server_negotiation.cc:922] calling sasl_server_start()
0929 22:50:34.685409 (+ 47us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 22:50:34.685425 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 22:50:34.685497 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0929 22:50:34.685553 (+ 56us) server_negotiation.cc:299] Negotiation successful
0929 22:50:34.685641 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":106,"thread_start_us":73,"threads_started":1}
I20260929 22:50:34.687059 14927 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 22:50:34.693326 20313 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 22:50:34.693828 20313 catalog_manager.cc:1504] Loading token signing keys...
I20260929 22:50:34.703579 20313 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4248c11a8feb41478da5498baf50b3ed: Generated new TSK 0
I20260929 22:50:34.703776 20313 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 22:50:34.713865 20324 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:50:34.710143 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33720 (local address 127.14.147.254:38225)
0929 22:50:34.710190 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0929 22:50:34.710192 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 22:50:34.710278 (+ 86us) server_negotiation.cc:407] Connection header received
0929 22:50:34.710335 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 22:50:34.710336 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 22:50:34.710348 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 22:50:34.710429 (+ 81us) server_negotiation.cc:226] Negotiated authn=JWT
0929 22:50:34.710534 (+ 105us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:50:34.711326 (+ 792us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:50:34.712152 (+ 826us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:50:34.712480 (+ 328us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:50:34.712547 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 22:50:34.712676 (+ 129us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0929 22:50:34.713646 (+ 970us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:38061/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0929 22:50:34.713775 (+ 129us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:33720: Failed to load JWKS: Error downloading JWKS from 'https://localhost:38061/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"server-negotiator.queue_time_us":25}
W20260929 22:50:34.713929 20324 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:33720: Failed to load JWKS: Error downloading JWKS from 'https://localhost:38061/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
W20260929 22:50:34.715672 20330 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 22:50:34.710051 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.147.254:38225 (local address 127.0.0.1:33720)
0929 22:50:34.710216 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0929 22:50:34.710226 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0929 22:50:34.710279 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 22:50:34.710426 (+ 147us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 22:50:34.710426 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 22:50:34.710428 (+ 2us) client_negotiation.cc:189] Negotiated authn=JWT
0929 22:50:34.710482 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 22:50:34.710483 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 22:50:34.711460 (+ 977us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 22:50:34.711461 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 22:50:34.712073 (+ 612us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 22:50:34.712075 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 22:50:34.712547 (+ 472us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 22:50:34.712548 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 22:50:34.712585 (+ 37us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 22:50:34.712590 (+ 5us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0929 22:50:34.713751 (+ 1161us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:38061/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0929 22:50:34.715574 (+ 1823us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.14.147.254:38225: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:38061/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"client-negotiator.queue_time_us":139,"mutex_wait_us":1603,"thread_start_us":95,"threads_started":1}
I20260929 22:50:34.716001 14927 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhWOCLk/build/release/bin/kudu with pid 20234
E20260929 22:50:34.724898 20331 webserver.cc:533] Webserver: sslize failed:
E20260929 22:50:34.724977 20331 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
2026-09-29T22:50:34Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithUntrustedCert (216 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS
2026-09-29T22:50:34Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T22:50:34Z Disabled control of system clock
I20260929 22:50:34.760004 14927 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhWOCLk/build/release/bin/kudu
/tmp/dist-test-taskhWOCLk/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790722213368738-14927-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790722213368738-14927-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790722213368738-14927-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790722213368738-14927-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.147.254:44845
--webserver_interface=127.14.147.254
--webserver_port=0
--builtin_ntp_servers=127.14.147.212:37095
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.147.254:44845
--enable_jwt_token_auth=true
--jwks_url=default.url
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790722213368738-14927-0/testchainca.cert with env {}
W20260929 22:50:34.839969 20335 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 22:50:34.840173 20335 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 22:50:34.840195 20335 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 22:50:34.841420 20335 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 22:50:34.841466 20335 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 22:50:34.841476 20335 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 22:50:34.841488 20335 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 22:50:34.841498 20335 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260929 22:50:34.841513 20335 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260929 22:50:34.841535 20335 flags.cc:425] Enabled experimental flag: --jwks_url=default.url
I20260929 22:50:34.843065 20335 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.147.212:37095
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790722213368738-14927-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790722213368738-14927-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.14.147.254:44845
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.147.254:44845
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=default.url
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790722213368738-14927-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.14.147.254
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790722213368738-14927-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20335
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790722213368738-14927-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 21:58:54 UTC on ccycloud.quasar-xguoti.root.comops.site
build id 4681390
I20260929 22:50:34.843312 20335 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 22:50:34.843531 20335 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 22:50:34.847970 20343 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 22:50:34.847976 20340 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 22:50:34.848097 20335 server_base.cc:1034] running on GCE node
W20260929 22:50:34.847982 20341 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 22:50:34.848420 20335 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 22:50:34.848665 20335 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 22:50:34.849799 20335 hybrid_clock.cc:648] HybridClock initialized: now 1790722234849782 us; error 32 us; skew 500 ppm
I20260929 22:50:34.850945 20335 webserver.cc:460] Webserver started at http://127.14.147.254:37763/ using document root <none> and password file <none>
I20260929 22:50:34.851136 20335 fs_manager.cc:359] Metadata directory not provided
I20260929 22:50:34.851176 20335 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 22:50:34.851272 20335 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 22:50:34.852352 20335 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790722213368738-14927-0/minicluster-data/master-0/data/instance:
uuid: "c7f866d5eeb44a758f5a707c496a8f2f"
format_stamp: "Formatted at 2026-09-29 22:50:34 on dist-test-slave-98mk"
server_key: "b0ae2885d34ec1b9e2765a0c3663adec"
server_key_iv: "a7d3cfcadef9326b83c784e12660aabd"
server_key_version: "encryptionkey@0"
I20260929 22:50:34.852685 20335 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790722213368738-14927-0/minicluster-data/master-0/wal/instance:
uuid: "c7f866d5eeb44a758f5a707c496a8f2f"
format_stamp: "Formatted at 2026-09-29 22:50:34 on dist-test-slave-98mk"
server_key: "b0ae2885d34ec1b9e2765a0c3663adec"
server_key_iv: "a7d3cfcadef9326b83c784e12660aabd"
server_key_version: "encryptionkey@0"
I20260929 22:50:34.853942 20335 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260929 22:50:34.854578 20349 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 22:50:34.854723 20335 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260929 22:50:34.854797 20335 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790722213368738-14927-0/minicluster-data/master-0/data,/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790722213368738-14927-0/minicluster-data/master-0/wal
uuid: "c7f866d5eeb44a758f5a707c496a8f2f"
format_stamp: "Formatted at 2026-09-29 22:50:34 on dist-test-slave-98mk"
server_key: "b0ae2885d34ec1b9e2765a0c3663adec"
server_key_iv: "a7d3cfcadef9326b83c784e12660aabd"
server_key_version: "encryptionkey@0"
I20260929 22:50:34.854871 20335 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790722213368738-14927-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790722213368738-14927-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790722213368738-14927-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 22:50:34.876705 20335 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 22:50:34.877022 20335 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 22:50:34.877141 20335 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 22:50:34.880780 20335 rpc_server.cc:307] RPC server started. Bound to: 127.14.147.254:44845
I20260929 22:50:34.880813 20401 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.147.254:44845 every 8 connection(s)
I20260929 22:50:34.881119 20335 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790722213368738-14927-0/minicluster-data/master-0/data/info.pb
I20260929 22:50:34.882025 20402 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 22:50:34.884322 20402 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c7f866d5eeb44a758f5a707c496a8f2f: Bootstrap starting.
I20260929 22:50:34.884330 14927 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhWOCLk/build/release/bin/kudu as pid 20335
I20260929 22:50:34.884413 14927 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790722213368738-14927-0/minicluster-data/master-0/wal/instance
I20260929 22:50:34.884573 14927 external_mini_cluster.cc:1428] Setting key 9a8402aff964eb93c85c70261c4987c6
I20260929 22:50:34.884943 20402 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c7f866d5eeb44a758f5a707c496a8f2f: Neither blocks nor log segments found. Creating new log.
I20260929 22:50:34.885295 20402 log.cc:824] T 00000000000000000000000000000000 P c7f866d5eeb44a758f5a707c496a8f2f: Log is configured to *not* fsync() on all Append() calls
I20260929 22:50:34.885972 20402 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c7f866d5eeb44a758f5a707c496a8f2f: No bootstrap required, opened a new log
I20260929 22:50:34.888005 20402 raft_consensus.cc:348] T 00000000000000000000000000000000 P c7f866d5eeb44a758f5a707c496a8f2f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c7f866d5eeb44a758f5a707c496a8f2f" member_type: VOTER last_known_addr { host: "127.14.147.254" port: 44845 } }
I20260929 22:50:34.888185 20402 raft_consensus.cc:374] T 00000000000000000000000000000000 P c7f866d5eeb44a758f5a707c496a8f2f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 22:50:34.888213 20402 raft_consensus.cc:729] T 00000000000000000000000000000000 P c7f866d5eeb44a758f5a707c496a8f2f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c7f866d5eeb44a758f5a707c496a8f2f, State: Initialized, Role: FOLLOWER
I20260929 22:50:34.888340 20402 consensus_queue.cc:260] T 00000000000000000000000000000000 P c7f866d5eeb44a758f5a707c496a8f2f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c7f866d5eeb44a758f5a707c496a8f2f" member_type: VOTER last_known_addr { host: "127.14.147.254" port: 44845 } }
I20260929 22:50:34.888422 20402 raft_consensus.cc:388] T 00000000000000000000000000000000 P c7f866d5eeb44a758f5a707c496a8f2f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 22:50:34.888445 20402 raft_consensus.cc:482] T 00000000000000000000000000000000 P c7f866d5eeb44a758f5a707c496a8f2f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 22:50:34.888490 20402 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c7f866d5eeb44a758f5a707c496a8f2f [term 0 FOLLOWER]: Advancing to term 1
I20260929 22:50:34.889171 20402 raft_consensus.cc:504] T 00000000000000000000000000000000 P c7f866d5eeb44a758f5a707c496a8f2f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c7f866d5eeb44a758f5a707c496a8f2f" member_type: VOTER last_known_addr { host: "127.14.147.254" port: 44845 } }
I20260929 22:50:34.889289 20402 leader_election.cc:302] T 00000000000000000000000000000000 P c7f866d5eeb44a758f5a707c496a8f2f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c7f866d5eeb44a758f5a707c496a8f2f; no voters:
I20260929 22:50:34.889453 20402 leader_election.cc:288] T 00000000000000000000000000000000 P c7f866d5eeb44a758f5a707c496a8f2f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 22:50:34.889484 20407 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c7f866d5eeb44a758f5a707c496a8f2f [term 1 FOLLOWER]: Leader election won for term 1
I20260929 22:50:34.889576 20407 raft_consensus.cc:686] T 00000000000000000000000000000000 P c7f866d5eeb44a758f5a707c496a8f2f [term 1 LEADER]: Becoming Leader. State: Replica: c7f866d5eeb44a758f5a707c496a8f2f, State: Running, Role: LEADER
I20260929 22:50:34.889699 20402 sys_catalog.cc:564] T 00000000000000000000000000000000 P c7f866d5eeb44a758f5a707c496a8f2f [sys.catalog]: configured and running, proceeding with master startup.
I20260929 22:50:34.889663 20407 consensus_queue.cc:237] T 00000000000000000000000000000000 P c7f866d5eeb44a758f5a707c496a8f2f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c7f866d5eeb44a758f5a707c496a8f2f" member_type: VOTER last_known_addr { host: "127.14.147.254" port: 44845 } }
I20260929 22:50:34.890020 20405 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:50:34.885039 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44734 (local address 127.14.147.254:44845)
0929 22:50:34.885234 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0929 22:50:34.885241 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 22:50:34.885267 (+ 26us) server_negotiation.cc:407] Connection header received
0929 22:50:34.885705 (+ 438us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 22:50:34.885712 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 22:50:34.885770 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 22:50:34.885888 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0929 22:50:34.886102 (+ 214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:50:34.886983 (+ 881us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:50:34.888068 (+ 1085us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:50:34.888552 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:50:34.888646 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 22:50:34.889228 (+ 582us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 22:50:34.889231 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 22:50:34.889232 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 22:50:34.889438 (+ 206us) server_negotiation.cc:922] calling sasl_server_start()
0929 22:50:34.889471 (+ 33us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 22:50:34.889482 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 22:50:34.889568 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0929 22:50:34.889677 (+ 109us) server_negotiation.cc:299] Negotiation successful
0929 22:50:34.889794 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":121,"thread_start_us":80,"threads_started":1}
I20260929 22:50:34.890100 20407 sys_catalog.cc:455] T 00000000000000000000000000000000 P c7f866d5eeb44a758f5a707c496a8f2f [sys.catalog]: SysCatalogTable state changed. Reason: New leader c7f866d5eeb44a758f5a707c496a8f2f. Latest consensus state: current_term: 1 leader_uuid: "c7f866d5eeb44a758f5a707c496a8f2f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c7f866d5eeb44a758f5a707c496a8f2f" member_type: VOTER last_known_addr { host: "127.14.147.254" port: 44845 } } }
I20260929 22:50:34.890194 20407 sys_catalog.cc:458] T 00000000000000000000000000000000 P c7f866d5eeb44a758f5a707c496a8f2f [sys.catalog]: This master's current role is: LEADER
I20260929 22:50:34.890393 20414 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 22:50:34.890893 20414 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 22:50:34.890929 14927 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 22:50:34.891516 20408 sys_catalog.cc:455] T 00000000000000000000000000000000 P c7f866d5eeb44a758f5a707c496a8f2f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c7f866d5eeb44a758f5a707c496a8f2f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c7f866d5eeb44a758f5a707c496a8f2f" member_type: VOTER last_known_addr { host: "127.14.147.254" port: 44845 } } }
I20260929 22:50:34.891593 20408 sys_catalog.cc:458] T 00000000000000000000000000000000 P c7f866d5eeb44a758f5a707c496a8f2f [sys.catalog]: This master's current role is: LEADER
I20260929 22:50:34.892403 20414 catalog_manager.cc:1347] Generated new cluster ID: 71b2d3b8d8344ead847248ebeb8e6093
I20260929 22:50:34.892452 20414 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 22:50:34.906472 20414 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 22:50:34.907001 20414 catalog_manager.cc:1504] Loading token signing keys...
I20260929 22:50:34.920346 20414 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c7f866d5eeb44a758f5a707c496a8f2f: Generated new TSK 0
I20260929 22:50:34.920554 20414 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 22:50:35.409742 20405 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:50:34.932911 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44746 (local address 127.14.147.254:44845)
0929 22:50:34.933265 (+ 354us) server_negotiation.cc:206] Beginning negotiation
0929 22:50:34.933267 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 22:50:34.933276 (+ 9us) server_negotiation.cc:407] Connection header received
0929 22:50:34.933320 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 22:50:34.933321 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 22:50:34.933334 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 22:50:34.933404 (+ 70us) server_negotiation.cc:226] Negotiated authn=JWT
0929 22:50:34.933568 (+ 164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:50:34.934283 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:50:34.935460 (+ 1177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:50:34.935695 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:50:34.935757 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 22:50:34.935954 (+ 197us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0929 22:50:35.409519 (+473565us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0929 22:50:35.409680 (+ 161us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:44746: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"server-negotiator.queue_time_us":330}
W20260929 22:50:35.409809 20405 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:44746: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
W20260929 22:50:35.409934 20431 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 22:50:34.932875 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.147.254:44845 (local address 127.0.0.1:44746)
0929 22:50:34.933064 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0929 22:50:34.933076 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0929 22:50:34.933157 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 22:50:34.933412 (+ 255us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 22:50:34.933413 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 22:50:34.933416 (+ 3us) client_negotiation.cc:189] Negotiated authn=JWT
0929 22:50:34.933482 (+ 66us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 22:50:34.933484 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 22:50:34.934364 (+ 880us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 22:50:34.934365 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 22:50:34.935330 (+ 965us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 22:50:34.935333 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 22:50:34.935778 (+ 445us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 22:50:34.935779 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 22:50:34.935845 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 22:50:34.935852 (+ 7us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0929 22:50:35.409685 (+473833us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0929 22:50:35.409825 (+ 140us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.14.147.254:44845: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"client-negotiator.queue_time_us":161,"thread_start_us":84,"threads_started":1}
I20260929 22:50:35.410276 14927 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhWOCLk/build/release/bin/kudu with pid 20335
2026-09-29T22:50:35Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS (674 ms)
[ RUN ] SecurityITest.TestWorldReadableKeytab
[ OK ] SecurityITest.TestWorldReadableKeytab (90 ms)
[ RUN ] SecurityITest.TestWorldReadablePrivateKey
[ OK ] SecurityITest.TestWorldReadablePrivateKey (86 ms)
[ RUN ] SecurityITest.TestCorruptKerberosCC
Loading random data
Initializing database '/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790722213368738-14927-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 22:50:35 dist-test-slave-98mk krb5kdc[20443](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 22:50:35 dist-test-slave-98mk krb5kdc[20443](info): set up 2 sockets
Sep 29 22:50:35 dist-test-slave-98mk krb5kdc[20443](info): commencing operation
krb5kdc: starting...
W20260929 22:50:36.536060 14927 mini_kdc.cc:121] Time spent starting KDC: real 0.931s user 0.000s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790722213368738-14927-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790722213368738-14927-0/krb5kdc/test-user.keytab.
Sep 29 22:50:36 dist-test-slave-98mk krb5kdc[20443](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790722236, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T22:50:36Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T22:50:36Z Disabled control of system clock
WARNING: no policy specified for kudu/127.14.147.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.147.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.147.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790722213368738-14927-0/krb5kdc/kudu_127.14.147.254.keytab.
Entry for principal kudu/127.14.147.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790722213368738-14927-0/krb5kdc/kudu_127.14.147.254.keytab.
I20260929 22:50:36.646303 14927 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhWOCLk/build/release/bin/kudu
/tmp/dist-test-taskhWOCLk/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790722213368738-14927-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790722213368738-14927-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790722213368738-14927-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790722213368738-14927-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.147.254:42323
--webserver_interface=127.14.147.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790722213368738-14927-0/krb5kdc/kudu_127.14.147.254.keytab
--principal=kudu/127.14.147.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.147.212:36273
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.147.254:42323
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790722213368738-14927-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790722213368738-14927-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790722213368738-14927-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 22:50:36.728765 20456 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 22:50:36.728992 20456 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 22:50:36.729017 20456 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 22:50:36.730268 20456 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 22:50:36.730321 20456 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 22:50:36.730334 20456 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 22:50:36.730346 20456 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 22:50:36.730367 20456 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 22:50:36.731986 20456 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.147.212:36273
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790722213368738-14927-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790722213368738-14927-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.14.147.254:42323
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790722213368738-14927-0/krb5kdc/kudu_127.14.147.254.keytab
--principal=kudu/127.14.147.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.147.254:42323
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790722213368738-14927-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.14.147.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20456
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790722213368738-14927-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 21:58:54 UTC on ccycloud.quasar-xguoti.root.comops.site
build id 4681390
I20260929 22:50:36.732241 20456 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 22:50:36.732472 20456 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 22:50:36.737389 20461 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 22:50:36.737367 20464 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 22:50:36.737564 20456 server_base.cc:1034] running on GCE node
W20260929 22:50:36.737363 20462 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 22:50:36.737866 20456 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 22:50:36.738157 20456 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 22:50:36.739298 20456 hybrid_clock.cc:648] HybridClock initialized: now 1790722236739285 us; error 30 us; skew 500 ppm
Sep 29 22:50:36 dist-test-slave-98mk krb5kdc[20443](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790722236, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.147.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 22:50:36.750224 20456 init.cc:377] Logged in from keytab as kudu/127.14.147.254@KRBTEST.COM (short username kudu)
I20260929 22:50:36.750721 20456 webserver.cc:460] Webserver started at http://127.14.147.254:38207/ using document root <none> and password file <none>
I20260929 22:50:36.750989 20456 fs_manager.cc:359] Metadata directory not provided
I20260929 22:50:36.751037 20456 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 22:50:36.751151 20456 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 22:50:36.752297 20456 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790722213368738-14927-0/minicluster-data/master-0/data/instance:
uuid: "3c8280969aef4a36b643551b559f7f19"
format_stamp: "Formatted at 2026-09-29 22:50:36 on dist-test-slave-98mk"
server_key: "86962dc9d4c7c69978776e3403de646e"
server_key_iv: "d676718af72e36c87abe40f8e5b4d250"
server_key_version: "encryptionkey@0"
I20260929 22:50:36.752624 20456 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790722213368738-14927-0/minicluster-data/master-0/wal/instance:
uuid: "3c8280969aef4a36b643551b559f7f19"
format_stamp: "Formatted at 2026-09-29 22:50:36 on dist-test-slave-98mk"
server_key: "86962dc9d4c7c69978776e3403de646e"
server_key_iv: "d676718af72e36c87abe40f8e5b4d250"
server_key_version: "encryptionkey@0"
I20260929 22:50:36.753926 20456 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260929 22:50:36.754611 20471 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 22:50:36.754773 20456 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260929 22:50:36.754854 20456 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790722213368738-14927-0/minicluster-data/master-0/data,/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790722213368738-14927-0/minicluster-data/master-0/wal
uuid: "3c8280969aef4a36b643551b559f7f19"
format_stamp: "Formatted at 2026-09-29 22:50:36 on dist-test-slave-98mk"
server_key: "86962dc9d4c7c69978776e3403de646e"
server_key_iv: "d676718af72e36c87abe40f8e5b4d250"
server_key_version: "encryptionkey@0"
I20260929 22:50:36.754913 20456 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790722213368738-14927-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790722213368738-14927-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790722213368738-14927-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 22:50:36.765201 20456 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 22:50:36.767239 20456 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 22:50:36.767356 20456 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 22:50:36.771340 20456 rpc_server.cc:307] RPC server started. Bound to: 127.14.147.254:42323
I20260929 22:50:36.771390 20523 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.147.254:42323 every 8 connection(s)
I20260929 22:50:36.771750 20456 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790722213368738-14927-0/minicluster-data/master-0/data/info.pb
I20260929 22:50:36.772608 20524 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 22:50:36.774871 20524 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3c8280969aef4a36b643551b559f7f19: Bootstrap starting.
I20260929 22:50:36.775491 20524 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3c8280969aef4a36b643551b559f7f19: Neither blocks nor log segments found. Creating new log.
I20260929 22:50:36.775753 20524 log.cc:824] T 00000000000000000000000000000000 P 3c8280969aef4a36b643551b559f7f19: Log is configured to *not* fsync() on all Append() calls
I20260929 22:50:36.776666 20524 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3c8280969aef4a36b643551b559f7f19: No bootstrap required, opened a new log
I20260929 22:50:36.778285 20524 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3c8280969aef4a36b643551b559f7f19 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3c8280969aef4a36b643551b559f7f19" member_type: VOTER last_known_addr { host: "127.14.147.254" port: 42323 } }
I20260929 22:50:36.778437 20524 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3c8280969aef4a36b643551b559f7f19 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 22:50:36.778481 20524 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3c8280969aef4a36b643551b559f7f19 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3c8280969aef4a36b643551b559f7f19, State: Initialized, Role: FOLLOWER
I20260929 22:50:36.778626 20524 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3c8280969aef4a36b643551b559f7f19 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3c8280969aef4a36b643551b559f7f19" member_type: VOTER last_known_addr { host: "127.14.147.254" port: 42323 } }
I20260929 22:50:36.778702 20524 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3c8280969aef4a36b643551b559f7f19 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 22:50:36.778735 20524 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3c8280969aef4a36b643551b559f7f19 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 22:50:36.778784 20524 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3c8280969aef4a36b643551b559f7f19 [term 0 FOLLOWER]: Advancing to term 1
I20260929 22:50:36.779330 20524 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3c8280969aef4a36b643551b559f7f19 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3c8280969aef4a36b643551b559f7f19" member_type: VOTER last_known_addr { host: "127.14.147.254" port: 42323 } }
I20260929 22:50:36.779460 20524 leader_election.cc:302] T 00000000000000000000000000000000 P 3c8280969aef4a36b643551b559f7f19 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3c8280969aef4a36b643551b559f7f19; no voters:
I20260929 22:50:36.779630 20524 leader_election.cc:288] T 00000000000000000000000000000000 P 3c8280969aef4a36b643551b559f7f19 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 22:50:36.779678 20527 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3c8280969aef4a36b643551b559f7f19 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 22:50:36.779896 20524 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3c8280969aef4a36b643551b559f7f19 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 22:50:36.779997 20527 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3c8280969aef4a36b643551b559f7f19 [term 1 LEADER]: Becoming Leader. State: Replica: 3c8280969aef4a36b643551b559f7f19, State: Running, Role: LEADER
I20260929 22:50:36.780097 20527 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3c8280969aef4a36b643551b559f7f19 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3c8280969aef4a36b643551b559f7f19" member_type: VOTER last_known_addr { host: "127.14.147.254" port: 42323 } }
I20260929 22:50:36.780429 20528 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3c8280969aef4a36b643551b559f7f19 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3c8280969aef4a36b643551b559f7f19" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3c8280969aef4a36b643551b559f7f19" member_type: VOTER last_known_addr { host: "127.14.147.254" port: 42323 } } }
I20260929 22:50:36.780514 20528 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3c8280969aef4a36b643551b559f7f19 [sys.catalog]: This master's current role is: LEADER
I20260929 22:50:36.780666 20529 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3c8280969aef4a36b643551b559f7f19 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3c8280969aef4a36b643551b559f7f19. Latest consensus state: current_term: 1 leader_uuid: "3c8280969aef4a36b643551b559f7f19" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3c8280969aef4a36b643551b559f7f19" member_type: VOTER last_known_addr { host: "127.14.147.254" port: 42323 } } }
I20260929 22:50:36.780692 20536 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 22:50:36.780717 20529 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3c8280969aef4a36b643551b559f7f19 [sys.catalog]: This master's current role is: LEADER
I20260929 22:50:36.781176 20536 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 22:50:36.781363 14927 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhWOCLk/build/release/bin/kudu as pid 20456
I20260929 22:50:36.781464 14927 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790722213368738-14927-0/minicluster-data/master-0/wal/instance
I20260929 22:50:36.781651 14927 external_mini_cluster.cc:1428] Setting key acbc07e3feedecb3525d441e29f44e44
I20260929 22:50:36.782807 20536 catalog_manager.cc:1347] Generated new cluster ID: 22e90226329746bebea7e996d3208c28
I20260929 22:50:36.782859 20536 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 29 22:50:36 dist-test-slave-98mk krb5kdc[20443](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790722236, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.147.254@KRBTEST.COM
I20260929 22:50:36.790848 20546 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:50:36.782097 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55100 (local address 127.14.147.254:42323)
0929 22:50:36.782249 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0929 22:50:36.782258 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 22:50:36.782375 (+ 117us) server_negotiation.cc:407] Connection header received
0929 22:50:36.782520 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 22:50:36.782525 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 22:50:36.782569 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 22:50:36.782690 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0929 22:50:36.783323 (+ 633us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:50:36.784286 (+ 963us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:50:36.785009 (+ 723us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:50:36.785447 (+ 438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:50:36.785539 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 22:50:36.787814 (+ 2275us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 22:50:36.787818 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 22:50:36.787818 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 22:50:36.787831 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0929 22:50:36.789383 (+ 1552us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 22:50:36.789707 (+ 324us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 22:50:36.789711 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 22:50:36.789711 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 22:50:36.789799 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 22:50:36.789967 (+ 168us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 22:50:36.789968 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 22:50:36.789968 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 22:50:36.790228 (+ 260us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 22:50:36.790377 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0929 22:50:36.790482 (+ 105us) server_negotiation.cc:299] Negotiation successful
0929 22:50:36.790634 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":104,"thread_start_us":77,"threads_started":1}
I20260929 22:50:36.808502 20536 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 22:50:36.809039 20536 catalog_manager.cc:1504] Loading token signing keys...
I20260929 22:50:36.812310 20536 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3c8280969aef4a36b643551b559f7f19: Generated new TSK 0
I20260929 22:50:36.812512 20536 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.14.147.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.147.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.147.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790722213368738-14927-0/krb5kdc/kudu_127.14.147.193.keytab.
Entry for principal kudu/127.14.147.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790722213368738-14927-0/krb5kdc/kudu_127.14.147.193.keytab.
I20260929 22:50:36.840317 14927 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhWOCLk/build/release/bin/kudu
/tmp/dist-test-taskhWOCLk/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790722213368738-14927-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790722213368738-14927-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790722213368738-14927-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790722213368738-14927-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.147.193:0
--local_ip_for_outbound_sockets=127.14.147.193
--webserver_interface=127.14.147.193
--webserver_port=0
--tserver_master_addrs=127.14.147.254:42323
--keytab_file=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790722213368738-14927-0/krb5kdc/kudu_127.14.147.193.keytab
--principal=kudu/127.14.147.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.147.212:36273
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790722213368738-14927-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790722213368738-14927-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790722213368738-14927-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 22:50:36.922490 20550 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 22:50:36.922725 20550 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 22:50:36.922752 20550 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 22:50:36.924072 20550 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 22:50:36.924125 20550 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 22:50:36.924154 20550 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.147.193
I20260929 22:50:36.925812 20550 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.147.212:36273
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790722213368738-14927-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790722213368738-14927-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790722213368738-14927-0/krb5kdc/kudu_127.14.147.193.keytab
--principal=kudu/127.14.147.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.147.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790722213368738-14927-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.14.147.193
--webserver_port=0
--tserver_master_addrs=127.14.147.254:42323
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20550
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.147.193
--log_dir=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790722213368738-14927-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 21:58:54 UTC on ccycloud.quasar-xguoti.root.comops.site
build id 4681390
I20260929 22:50:36.926077 20550 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 22:50:36.926311 20550 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 22:50:36.931246 20558 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 22:50:36.931246 20555 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 22:50:36.931389 20556 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 22:50:36.931825 20550 server_base.cc:1034] running on GCE node
I20260929 22:50:36.932134 20550 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 22:50:36.932466 20550 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 22:50:36.933673 20550 hybrid_clock.cc:648] HybridClock initialized: now 1790722236933630 us; error 64 us; skew 500 ppm
Sep 29 22:50:36 dist-test-slave-98mk krb5kdc[20443](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790722236, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.147.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 22:50:36.936349 20550 init.cc:377] Logged in from keytab as kudu/127.14.147.193@KRBTEST.COM (short username kudu)
I20260929 22:50:36.936818 20550 webserver.cc:460] Webserver started at http://127.14.147.193:44331/ using document root <none> and password file <none>
I20260929 22:50:36.937047 20550 fs_manager.cc:359] Metadata directory not provided
I20260929 22:50:36.937098 20550 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 22:50:36.937230 20550 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 22:50:36.938282 20550 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790722213368738-14927-0/minicluster-data/ts-0/data/instance:
uuid: "f1f961c408d84deba6b8d03226010a33"
format_stamp: "Formatted at 2026-09-29 22:50:36 on dist-test-slave-98mk"
server_key: "7aaa8b1821e40b0bff7195977e41a360"
server_key_iv: "5e76f3c970d7e77df1a80aef9d78a170"
server_key_version: "encryptionkey@0"
I20260929 22:50:36.938621 20550 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790722213368738-14927-0/minicluster-data/ts-0/wal/instance:
uuid: "f1f961c408d84deba6b8d03226010a33"
format_stamp: "Formatted at 2026-09-29 22:50:36 on dist-test-slave-98mk"
server_key: "7aaa8b1821e40b0bff7195977e41a360"
server_key_iv: "5e76f3c970d7e77df1a80aef9d78a170"
server_key_version: "encryptionkey@0"
I20260929 22:50:36.939990 20550 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260929 22:50:36.940708 20565 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 22:50:36.940879 20550 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260929 22:50:36.940974 20550 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790722213368738-14927-0/minicluster-data/ts-0/data,/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790722213368738-14927-0/minicluster-data/ts-0/wal
uuid: "f1f961c408d84deba6b8d03226010a33"
format_stamp: "Formatted at 2026-09-29 22:50:36 on dist-test-slave-98mk"
server_key: "7aaa8b1821e40b0bff7195977e41a360"
server_key_iv: "5e76f3c970d7e77df1a80aef9d78a170"
server_key_version: "encryptionkey@0"
I20260929 22:50:36.941043 20550 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790722213368738-14927-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790722213368738-14927-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790722213368738-14927-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 22:50:36.957439 20550 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 22:50:36.959620 20550 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 22:50:36.959754 20550 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 22:50:36.959982 20550 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 22:50:36.960260 20550 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 22:50:36.960292 20550 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 22:50:36.960322 20550 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 22:50:36.960346 20550 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 22:50:36.965869 20550 rpc_server.cc:307] RPC server started. Bound to: 127.14.147.193:33305
I20260929 22:50:36.965924 20678 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.147.193:33305 every 8 connection(s)
I20260929 22:50:36.966228 20550 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790722213368738-14927-0/minicluster-data/ts-0/data/info.pb
Sep 29 22:50:36 dist-test-slave-98mk krb5kdc[20443](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790722236, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.147.193@KRBTEST.COM for kudu/127.14.147.254@KRBTEST.COM
I20260929 22:50:36.975358 20546 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:50:36.966924 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.147.193:35667 (local address 127.14.147.254:42323)
0929 22:50:36.966994 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0929 22:50:36.966995 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0929 22:50:36.967190 (+ 195us) server_negotiation.cc:407] Connection header received
0929 22:50:36.968155 (+ 965us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 22:50:36.968156 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 22:50:36.968168 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 22:50:36.968233 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0929 22:50:36.968847 (+ 614us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:50:36.969346 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:50:36.970300 (+ 954us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:50:36.970484 (+ 184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:50:36.970528 (+ 44us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 22:50:36.972714 (+ 2186us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 22:50:36.972716 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 22:50:36.972717 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 22:50:36.972725 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0929 22:50:36.974026 (+ 1301us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 22:50:36.974449 (+ 423us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 22:50:36.974451 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 22:50:36.974451 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 22:50:36.974506 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 22:50:36.974723 (+ 217us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 22:50:36.974724 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 22:50:36.974724 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 22:50:36.974849 (+ 125us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 22:50:36.974904 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0929 22:50:36.975169 (+ 265us) server_negotiation.cc:299] Negotiation successful
0929 22:50:36.975264 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260929 22:50:36.975560 20681 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:50:36.966891 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.147.254:42323 (local address 127.14.147.193:35667)
0929 22:50:36.967064 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0929 22:50:36.967108 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0929 22:50:36.968067 (+ 959us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 22:50:36.968293 (+ 226us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 22:50:36.968296 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 22:50:36.968562 (+ 266us) client_negotiation.cc:189] Negotiated authn=SASL
0929 22:50:36.968754 (+ 192us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 22:50:36.968758 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 22:50:36.969422 (+ 664us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 22:50:36.969423 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 22:50:36.970235 (+ 812us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 22:50:36.970238 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 22:50:36.970537 (+ 299us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 22:50:36.970538 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 22:50:36.970638 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 22:50:36.971093 (+ 455us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 22:50:36.971100 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0929 22:50:36.972609 (+ 1509us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 22:50:36.974123 (+ 1514us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 22:50:36.974133 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 22:50:36.974136 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 22:50:36.974377 (+ 241us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 22:50:36.974561 (+ 184us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 22:50:36.974561 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 22:50:36.974562 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 22:50:36.974666 (+ 104us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 22:50:36.974903 (+ 237us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 22:50:36.974910 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 22:50:36.975049 (+ 139us) client_negotiation.cc:769] Sending connection context
0929 22:50:36.975140 (+ 91us) client_negotiation.cc:240] Negotiation successful
0929 22:50:36.975318 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":87,"threads_started":1}
I20260929 22:50:36.975574 14927 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhWOCLk/build/release/bin/kudu as pid 20550
I20260929 22:50:36.975715 14927 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790722213368738-14927-0/minicluster-data/ts-0/wal/instance
I20260929 22:50:36.975989 14927 external_mini_cluster.cc:1428] Setting key 5080a1320bce2121d55bbfbd546b894a
I20260929 22:50:36.976194 20679 heartbeater.cc:347] Connected to a master server at 127.14.147.254:42323
I20260929 22:50:36.976302 20679 heartbeater.cc:464] Registering TS with master...
I20260929 22:50:36.976528 20679 heartbeater.cc:511] Master 127.14.147.254:42323 requested a full tablet report, sending...
I20260929 22:50:36.977151 20488 ts_manager.cc:194] Registered new tserver with Master: f1f961c408d84deba6b8d03226010a33 (127.14.147.193:33305)
I20260929 22:50:36.977773 20488 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.147.193@KRBTEST.COM'} at 127.14.147.193:35667
WARNING: no policy specified for kudu/127.14.147.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.147.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.147.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790722213368738-14927-0/krb5kdc/kudu_127.14.147.194.keytab.
Entry for principal kudu/127.14.147.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790722213368738-14927-0/krb5kdc/kudu_127.14.147.194.keytab.
I20260929 22:50:37.002684 14927 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhWOCLk/build/release/bin/kudu
/tmp/dist-test-taskhWOCLk/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790722213368738-14927-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790722213368738-14927-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790722213368738-14927-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790722213368738-14927-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.147.194:0
--local_ip_for_outbound_sockets=127.14.147.194
--webserver_interface=127.14.147.194
--webserver_port=0
--tserver_master_addrs=127.14.147.254:42323
--keytab_file=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790722213368738-14927-0/krb5kdc/kudu_127.14.147.194.keytab
--principal=kudu/127.14.147.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.147.212:36273
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790722213368738-14927-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790722213368738-14927-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790722213368738-14927-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 22:50:37.084506 20684 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 22:50:37.084739 20684 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 22:50:37.084764 20684 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 22:50:37.086050 20684 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 22:50:37.086100 20684 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 22:50:37.086130 20684 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.147.194
I20260929 22:50:37.087898 20684 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.147.212:36273
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790722213368738-14927-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790722213368738-14927-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790722213368738-14927-0/krb5kdc/kudu_127.14.147.194.keytab
--principal=kudu/127.14.147.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.147.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790722213368738-14927-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.14.147.194
--webserver_port=0
--tserver_master_addrs=127.14.147.254:42323
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20684
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.147.194
--log_dir=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790722213368738-14927-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 21:58:54 UTC on ccycloud.quasar-xguoti.root.comops.site
build id 4681390
I20260929 22:50:37.088150 20684 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 22:50:37.088495 20684 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260929 22:50:37.093596 20684 server_base.cc:1034] running on GCE node
W20260929 22:50:37.093561 20689 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 22:50:37.093577 20692 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 22:50:37.093542 20690 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 22:50:37.093899 20684 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 22:50:37.094134 20684 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 22:50:37.095269 20684 hybrid_clock.cc:648] HybridClock initialized: now 1790722237095250 us; error 38 us; skew 500 ppm
Sep 29 22:50:37 dist-test-slave-98mk krb5kdc[20443](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790722237, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.147.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 22:50:37.097697 20684 init.cc:377] Logged in from keytab as kudu/127.14.147.194@KRBTEST.COM (short username kudu)
I20260929 22:50:37.098115 20684 webserver.cc:460] Webserver started at http://127.14.147.194:42265/ using document root <none> and password file <none>
I20260929 22:50:37.098317 20684 fs_manager.cc:359] Metadata directory not provided
I20260929 22:50:37.098359 20684 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 22:50:37.098454 20684 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 22:50:37.099561 20684 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790722213368738-14927-0/minicluster-data/ts-1/data/instance:
uuid: "fc58cb814d6440d48650a95be627160b"
format_stamp: "Formatted at 2026-09-29 22:50:37 on dist-test-slave-98mk"
server_key: "ed18d3f2b42d36307263f80ab02ed484"
server_key_iv: "ce1b066d7d8bf6796ef7bc87919c0881"
server_key_version: "encryptionkey@0"
I20260929 22:50:37.099889 20684 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790722213368738-14927-0/minicluster-data/ts-1/wal/instance:
uuid: "fc58cb814d6440d48650a95be627160b"
format_stamp: "Formatted at 2026-09-29 22:50:37 on dist-test-slave-98mk"
server_key: "ed18d3f2b42d36307263f80ab02ed484"
server_key_iv: "ce1b066d7d8bf6796ef7bc87919c0881"
server_key_version: "encryptionkey@0"
I20260929 22:50:37.101260 20684 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260929 22:50:37.102037 20699 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 22:50:37.102288 20684 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260929 22:50:37.102378 20684 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790722213368738-14927-0/minicluster-data/ts-1/data,/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790722213368738-14927-0/minicluster-data/ts-1/wal
uuid: "fc58cb814d6440d48650a95be627160b"
format_stamp: "Formatted at 2026-09-29 22:50:37 on dist-test-slave-98mk"
server_key: "ed18d3f2b42d36307263f80ab02ed484"
server_key_iv: "ce1b066d7d8bf6796ef7bc87919c0881"
server_key_version: "encryptionkey@0"
I20260929 22:50:37.102442 20684 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790722213368738-14927-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790722213368738-14927-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790722213368738-14927-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 22:50:37.108879 20684 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 22:50:37.110886 20684 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 22:50:37.110999 20684 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 22:50:37.111196 20684 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 22:50:37.111508 20684 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 22:50:37.111544 20684 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 22:50:37.111567 20684 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 22:50:37.111582 20684 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 22:50:37.117093 20684 rpc_server.cc:307] RPC server started. Bound to: 127.14.147.194:42399
I20260929 22:50:37.117138 20812 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.147.194:42399 every 8 connection(s)
I20260929 22:50:37.117446 20684 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790722213368738-14927-0/minicluster-data/ts-1/data/info.pb
Sep 29 22:50:37 dist-test-slave-98mk krb5kdc[20443](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790722237, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.147.194@KRBTEST.COM for kudu/127.14.147.254@KRBTEST.COM
I20260929 22:50:37.127184 20546 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:50:37.118272 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.147.194:57663 (local address 127.14.147.254:42323)
0929 22:50:37.118336 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0929 22:50:37.118338 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 22:50:37.118416 (+ 78us) server_negotiation.cc:407] Connection header received
0929 22:50:37.119392 (+ 976us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 22:50:37.119395 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 22:50:37.119407 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 22:50:37.119519 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0929 22:50:37.120182 (+ 663us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:50:37.120690 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:50:37.121714 (+ 1024us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:50:37.121919 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:50:37.121985 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 22:50:37.124229 (+ 2244us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 22:50:37.124231 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 22:50:37.124231 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 22:50:37.124239 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0929 22:50:37.125499 (+ 1260us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 22:50:37.125950 (+ 451us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 22:50:37.125951 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 22:50:37.125951 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 22:50:37.126010 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 22:50:37.126292 (+ 282us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 22:50:37.126292 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 22:50:37.126292 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 22:50:37.126645 (+ 353us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 22:50:37.126725 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0929 22:50:37.127028 (+ 303us) server_negotiation.cc:299] Negotiation successful
0929 22:50:37.127114 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260929 22:50:37.127331 20815 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:50:37.118125 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.147.254:42323 (local address 127.14.147.194:57663)
0929 22:50:37.118295 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0929 22:50:37.118340 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0929 22:50:37.119247 (+ 907us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 22:50:37.119592 (+ 345us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 22:50:37.119596 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 22:50:37.119871 (+ 275us) client_negotiation.cc:189] Negotiated authn=SASL
0929 22:50:37.120069 (+ 198us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 22:50:37.120073 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 22:50:37.120780 (+ 707us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 22:50:37.120781 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 22:50:37.121608 (+ 827us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 22:50:37.121611 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 22:50:37.122022 (+ 411us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 22:50:37.122023 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 22:50:37.122144 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 22:50:37.122688 (+ 544us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 22:50:37.122697 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0929 22:50:37.124142 (+ 1445us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 22:50:37.125588 (+ 1446us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 22:50:37.125596 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 22:50:37.125598 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 22:50:37.125851 (+ 253us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 22:50:37.126115 (+ 264us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 22:50:37.126116 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 22:50:37.126117 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 22:50:37.126238 (+ 121us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 22:50:37.126726 (+ 488us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 22:50:37.126732 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 22:50:37.126881 (+ 149us) client_negotiation.cc:769] Sending connection context
0929 22:50:37.126979 (+ 98us) client_negotiation.cc:240] Negotiation successful
0929 22:50:37.127117 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":71,"threads_started":1}
I20260929 22:50:37.127617 14927 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhWOCLk/build/release/bin/kudu as pid 20684
I20260929 22:50:37.127702 14927 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790722213368738-14927-0/minicluster-data/ts-1/wal/instance
I20260929 22:50:37.127897 14927 external_mini_cluster.cc:1428] Setting key c732f9d89e071c1a5849d2209a04feae
I20260929 22:50:37.127931 20813 heartbeater.cc:347] Connected to a master server at 127.14.147.254:42323
I20260929 22:50:37.128032 20813 heartbeater.cc:464] Registering TS with master...
I20260929 22:50:37.128226 20813 heartbeater.cc:511] Master 127.14.147.254:42323 requested a full tablet report, sending...
I20260929 22:50:37.128609 20488 ts_manager.cc:194] Registered new tserver with Master: fc58cb814d6440d48650a95be627160b (127.14.147.194:42399)
I20260929 22:50:37.128990 20488 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.147.194@KRBTEST.COM'} at 127.14.147.194:57663
WARNING: no policy specified for kudu/127.14.147.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.147.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.147.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790722213368738-14927-0/krb5kdc/kudu_127.14.147.195.keytab.
Entry for principal kudu/127.14.147.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790722213368738-14927-0/krb5kdc/kudu_127.14.147.195.keytab.
I20260929 22:50:37.151396 14927 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhWOCLk/build/release/bin/kudu
/tmp/dist-test-taskhWOCLk/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790722213368738-14927-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790722213368738-14927-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790722213368738-14927-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790722213368738-14927-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.147.195:0
--local_ip_for_outbound_sockets=127.14.147.195
--webserver_interface=127.14.147.195
--webserver_port=0
--tserver_master_addrs=127.14.147.254:42323
--keytab_file=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790722213368738-14927-0/krb5kdc/kudu_127.14.147.195.keytab
--principal=kudu/127.14.147.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.147.212:36273
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790722213368738-14927-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790722213368738-14927-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790722213368738-14927-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 22:50:37.232582 20818 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 22:50:37.232803 20818 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 22:50:37.232829 20818 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 22:50:37.234028 20818 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 22:50:37.234074 20818 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 22:50:37.234105 20818 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.147.195
I20260929 22:50:37.235692 20818 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.147.212:36273
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790722213368738-14927-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790722213368738-14927-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790722213368738-14927-0/krb5kdc/kudu_127.14.147.195.keytab
--principal=kudu/127.14.147.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.147.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790722213368738-14927-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.14.147.195
--webserver_port=0
--tserver_master_addrs=127.14.147.254:42323
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20818
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.147.195
--log_dir=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790722213368738-14927-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 21:58:54 UTC on ccycloud.quasar-xguoti.root.comops.site
build id 4681390
I20260929 22:50:37.235944 20818 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 22:50:37.236159 20818 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 22:50:37.240845 20826 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 22:50:37.240872 20823 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 22:50:37.240870 20824 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 22:50:37.241070 20818 server_base.cc:1034] running on GCE node
I20260929 22:50:37.241274 20818 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 22:50:37.241499 20818 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 22:50:37.242663 20818 hybrid_clock.cc:648] HybridClock initialized: now 1790722237242642 us; error 30 us; skew 500 ppm
Sep 29 22:50:37 dist-test-slave-98mk krb5kdc[20443](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790722237, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.147.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 22:50:37.245252 20818 init.cc:377] Logged in from keytab as kudu/127.14.147.195@KRBTEST.COM (short username kudu)
I20260929 22:50:37.245697 20818 webserver.cc:460] Webserver started at http://127.14.147.195:36399/ using document root <none> and password file <none>
I20260929 22:50:37.245909 20818 fs_manager.cc:359] Metadata directory not provided
I20260929 22:50:37.245954 20818 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 22:50:37.246074 20818 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 22:50:37.247104 20818 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790722213368738-14927-0/minicluster-data/ts-2/data/instance:
uuid: "1a07df1c1ae34b0397d3c437c05a9971"
format_stamp: "Formatted at 2026-09-29 22:50:37 on dist-test-slave-98mk"
server_key: "3d44014bcebfc06d43e58c391428d997"
server_key_iv: "54aff7ccee9c0aaee66a069ba8ad9d23"
server_key_version: "encryptionkey@0"
I20260929 22:50:37.247473 20818 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790722213368738-14927-0/minicluster-data/ts-2/wal/instance:
uuid: "1a07df1c1ae34b0397d3c437c05a9971"
format_stamp: "Formatted at 2026-09-29 22:50:37 on dist-test-slave-98mk"
server_key: "3d44014bcebfc06d43e58c391428d997"
server_key_iv: "54aff7ccee9c0aaee66a069ba8ad9d23"
server_key_version: "encryptionkey@0"
I20260929 22:50:37.248696 20818 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260929 22:50:37.249378 20833 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 22:50:37.249527 20818 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260929 22:50:37.249605 20818 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790722213368738-14927-0/minicluster-data/ts-2/data,/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790722213368738-14927-0/minicluster-data/ts-2/wal
uuid: "1a07df1c1ae34b0397d3c437c05a9971"
format_stamp: "Formatted at 2026-09-29 22:50:37 on dist-test-slave-98mk"
server_key: "3d44014bcebfc06d43e58c391428d997"
server_key_iv: "54aff7ccee9c0aaee66a069ba8ad9d23"
server_key_version: "encryptionkey@0"
I20260929 22:50:37.249683 20818 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790722213368738-14927-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790722213368738-14927-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790722213368738-14927-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 22:50:37.277734 20818 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 22:50:37.280100 20818 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 22:50:37.280241 20818 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 22:50:37.280483 20818 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 22:50:37.280898 20818 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 22:50:37.280936 20818 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 22:50:37.280966 20818 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 22:50:37.280990 20818 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 22:50:37.286532 20818 rpc_server.cc:307] RPC server started. Bound to: 127.14.147.195:35225
I20260929 22:50:37.286576 20946 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.147.195:35225 every 8 connection(s)
I20260929 22:50:37.286895 20818 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790722213368738-14927-0/minicluster-data/ts-2/data/info.pb
Sep 29 22:50:37 dist-test-slave-98mk krb5kdc[20443](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790722237, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.147.195@KRBTEST.COM for kudu/127.14.147.254@KRBTEST.COM
I20260929 22:50:37.296547 14927 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhWOCLk/build/release/bin/kudu as pid 20818
I20260929 22:50:37.296635 14927 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790722213368738-14927-0/minicluster-data/ts-2/wal/instance
I20260929 22:50:37.296819 14927 external_mini_cluster.cc:1428] Setting key 176e2b61e495ea4769cfa6133e02f3bd
I20260929 22:50:37.296976 20546 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:50:37.287622 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.147.195:59699 (local address 127.14.147.254:42323)
0929 22:50:37.287674 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0929 22:50:37.287675 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0929 22:50:37.287905 (+ 230us) server_negotiation.cc:407] Connection header received
0929 22:50:37.288883 (+ 978us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 22:50:37.288884 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 22:50:37.288897 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 22:50:37.288965 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0929 22:50:37.289595 (+ 630us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:50:37.290119 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:50:37.291097 (+ 978us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:50:37.291281 (+ 184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:50:37.291336 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 22:50:37.294082 (+ 2746us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 22:50:37.294084 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 22:50:37.294084 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 22:50:37.294093 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0929 22:50:37.295529 (+ 1436us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 22:50:37.295989 (+ 460us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 22:50:37.295991 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 22:50:37.295991 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 22:50:37.296047 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 22:50:37.296270 (+ 223us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 22:50:37.296271 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 22:50:37.296271 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 22:50:37.296407 (+ 136us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 22:50:37.296464 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0929 22:50:37.296796 (+ 332us) server_negotiation.cc:299] Negotiation successful
0929 22:50:37.296889 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260929 22:50:37.297145 20949 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:50:37.287610 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.147.254:42323 (local address 127.14.147.195:59699)
0929 22:50:37.287786 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0929 22:50:37.287826 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0929 22:50:37.288767 (+ 941us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 22:50:37.289036 (+ 269us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 22:50:37.289040 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 22:50:37.289305 (+ 265us) client_negotiation.cc:189] Negotiated authn=SASL
0929 22:50:37.289508 (+ 203us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 22:50:37.289513 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 22:50:37.290217 (+ 704us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 22:50:37.290219 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 22:50:37.291013 (+ 794us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 22:50:37.291016 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 22:50:37.291361 (+ 345us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 22:50:37.291362 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 22:50:37.291515 (+ 153us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 22:50:37.292081 (+ 566us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 22:50:37.292094 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0929 22:50:37.293927 (+ 1833us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 22:50:37.295618 (+ 1691us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 22:50:37.295631 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 22:50:37.295633 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 22:50:37.295912 (+ 279us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 22:50:37.296102 (+ 190us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 22:50:37.296103 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 22:50:37.296103 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 22:50:37.296214 (+ 111us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 22:50:37.296463 (+ 249us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 22:50:37.296470 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 22:50:37.296614 (+ 144us) client_negotiation.cc:769] Sending connection context
0929 22:50:37.296742 (+ 128us) client_negotiation.cc:240] Negotiation successful
0929 22:50:37.296909 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":135,"thread_start_us":89,"threads_started":1}
I20260929 22:50:37.297561 20947 heartbeater.cc:347] Connected to a master server at 127.14.147.254:42323
I20260929 22:50:37.297672 20947 heartbeater.cc:464] Registering TS with master...
I20260929 22:50:37.297861 20947 heartbeater.cc:511] Master 127.14.147.254:42323 requested a full tablet report, sending...
I20260929 22:50:37.298225 20487 ts_manager.cc:194] Registered new tserver with Master: 1a07df1c1ae34b0397d3c437c05a9971 (127.14.147.195:35225)
I20260929 22:50:37.298583 20487 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.147.195@KRBTEST.COM'} at 127.14.147.195:59699
I20260929 22:50:37.308784 14927 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal test-admin with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790722213368738-14927-0/krb5kdc/test-admin.keytab.
Entry for principal test-admin with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790722213368738-14927-0/krb5kdc/test-admin.keytab.
Sep 29 22:50:37 dist-test-slave-98mk krb5kdc[20443](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790722237, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 22:50:37.322781 14927 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
W20260929 22:50:37.324321 20954 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 22:50:37.323635 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36834 (local address 127.14.147.193:33305)
0929 22:50:37.323817 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0929 22:50:37.323821 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 22:50:37.323831 (+ 10us) server_negotiation.cc:407] Connection header received
0929 22:50:37.323859 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 22:50:37.323861 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 22:50:37.323874 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 22:50:37.323942 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0929 22:50:37.324243 (+ 301us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36834: BlockingRecv error: recv got EOF from 127.0.0.1:36834 (error 108)
Metrics: {"server-negotiator.queue_time_us":147,"thread_start_us":107,"threads_started":1}
W20260929 22:50:37.324453 14927 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790722213368738-14927-0/krb5kdc/krb5cc' of type FILE
Sep 29 22:50:37 dist-test-slave-98mk krb5kdc[20443](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790722237, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 22:50:37.325367 14927 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 29 22:50:37 dist-test-slave-98mk krb5kdc[20443](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790722237, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 22:50:37.326170 14927 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 29 22:50:37 dist-test-slave-98mk krb5kdc[20443](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790722237, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.147.193@KRBTEST.COM
I20260929 22:50:37.334837 20954 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:50:37.326653 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36836 (local address 127.14.147.193:33305)
0929 22:50:37.326691 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0929 22:50:37.326693 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 22:50:37.326727 (+ 34us) server_negotiation.cc:407] Connection header received
0929 22:50:37.326792 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 22:50:37.326793 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 22:50:37.326804 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 22:50:37.326856 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0929 22:50:37.327250 (+ 394us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:50:37.328131 (+ 881us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:50:37.328962 (+ 831us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:50:37.329428 (+ 466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:50:37.329536 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 22:50:37.331651 (+ 2115us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 22:50:37.331660 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 22:50:37.331661 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 22:50:37.331669 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0929 22:50:37.333433 (+ 1764us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 22:50:37.333909 (+ 476us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 22:50:37.333912 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 22:50:37.333912 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 22:50:37.333976 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 22:50:37.334252 (+ 276us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 22:50:37.334253 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 22:50:37.334253 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 22:50:37.334428 (+ 175us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 22:50:37.334539 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0929 22:50:37.334665 (+ 126us) server_negotiation.cc:299] Negotiation successful
0929 22:50:37.334737 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
W20260929 22:50:37.336687 20954 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 22:50:37.336134 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36848 (local address 127.14.147.193:33305)
0929 22:50:37.336186 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0929 22:50:37.336188 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 22:50:37.336246 (+ 58us) server_negotiation.cc:407] Connection header received
0929 22:50:37.336295 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 22:50:37.336296 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 22:50:37.336305 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 22:50:37.336340 (+ 35us) server_negotiation.cc:226] Negotiated authn=SASL
0929 22:50:37.336639 (+ 299us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36848: BlockingRecv error: recv got EOF from 127.0.0.1:36848 (error 108)
Metrics: {"server-negotiator.queue_time_us":31}
W20260929 22:50:37.336733 14927 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790722213368738-14927-0/krb5kdc/krb5cc' of type FILE
Sep 29 22:50:37 dist-test-slave-98mk krb5kdc[20443](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790722237, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 22:50:37.337723 14927 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 29 22:50:37 dist-test-slave-98mk krb5kdc[20443](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790722237, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 22:50:37.338543 14927 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 29 22:50:37 dist-test-slave-98mk krb5kdc[20443](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790722237, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.147.193@KRBTEST.COM
I20260929 22:50:37.344848 20954 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:50:37.339091 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36862 (local address 127.14.147.193:33305)
0929 22:50:37.339139 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0929 22:50:37.339140 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0929 22:50:37.339148 (+ 8us) server_negotiation.cc:407] Connection header received
0929 22:50:37.339194 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 22:50:37.339194 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 22:50:37.339202 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 22:50:37.339245 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0929 22:50:37.339631 (+ 386us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:50:37.340127 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:50:37.340750 (+ 623us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:50:37.340946 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:50:37.340991 (+ 45us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 22:50:37.342838 (+ 1847us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 22:50:37.342839 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 22:50:37.342839 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 22:50:37.342845 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0929 22:50:37.343960 (+ 1115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 22:50:37.344313 (+ 353us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 22:50:37.344314 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 22:50:37.344314 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 22:50:37.344362 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 22:50:37.344495 (+ 133us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 22:50:37.344495 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 22:50:37.344496 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 22:50:37.344583 (+ 87us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 22:50:37.344658 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0929 22:50:37.344724 (+ 66us) server_negotiation.cc:299] Negotiation successful
0929 22:50:37.344785 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
W20260929 22:50:37.346113 20954 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 22:50:37.345700 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36876 (local address 127.14.147.193:33305)
0929 22:50:37.345757 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0929 22:50:37.345759 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 22:50:37.345766 (+ 7us) server_negotiation.cc:407] Connection header received
0929 22:50:37.345807 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 22:50:37.345808 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 22:50:37.345816 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 22:50:37.345855 (+ 39us) server_negotiation.cc:226] Negotiated authn=SASL
0929 22:50:37.346068 (+ 213us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36876: BlockingRecv error: recv got EOF from 127.0.0.1:36876 (error 108)
Metrics: {"server-negotiator.queue_time_us":38}
W20260929 22:50:37.346170 14927 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790722213368738-14927-0/krb5kdc/krb5cc' of type FILE
Sep 29 22:50:37 dist-test-slave-98mk krb5kdc[20443](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790722237, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 22:50:37.347088 14927 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 29 22:50:37 dist-test-slave-98mk krb5kdc[20443](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790722237, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 22:50:37.347924 14927 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 29 22:50:37 dist-test-slave-98mk krb5kdc[20443](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790722237, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.147.193@KRBTEST.COM
I20260929 22:50:37.354001 20954 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:50:37.348385 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36884 (local address 127.14.147.193:33305)
0929 22:50:37.348436 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0929 22:50:37.348438 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 22:50:37.348448 (+ 10us) server_negotiation.cc:407] Connection header received
0929 22:50:37.348510 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 22:50:37.348511 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 22:50:37.348519 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 22:50:37.348564 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0929 22:50:37.348869 (+ 305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:50:37.349326 (+ 457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:50:37.349951 (+ 625us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:50:37.350154 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:50:37.350204 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 22:50:37.352012 (+ 1808us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 22:50:37.352013 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 22:50:37.352014 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 22:50:37.352020 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0929 22:50:37.353059 (+ 1039us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 22:50:37.353335 (+ 276us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 22:50:37.353336 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 22:50:37.353337 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 22:50:37.353416 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 22:50:37.353575 (+ 159us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 22:50:37.353576 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 22:50:37.353576 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 22:50:37.353666 (+ 90us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 22:50:37.353745 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0929 22:50:37.353838 (+ 93us) server_negotiation.cc:299] Negotiation successful
0929 22:50:37.353909 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260929 22:50:37.354599 14927 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhWOCLk/build/release/bin/kudu with pid 20550
I20260929 22:50:37.359007 14927 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhWOCLk/build/release/bin/kudu with pid 20684
I20260929 22:50:37.363397 14927 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhWOCLk/build/release/bin/kudu with pid 20818
I20260929 22:50:37.366938 14927 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhWOCLk/build/release/bin/kudu with pid 20456
2026-09-29T22:50:37Z chronyd exiting
[ OK ] SecurityITest.TestCorruptKerberosCC (1780 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipalMultipleMaster
Loading random data
Initializing database '/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790722213368738-14927-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 22:50:37 dist-test-slave-98mk krb5kdc[20968](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 22:50:37 dist-test-slave-98mk krb5kdc[20968](info): set up 2 sockets
Sep 29 22:50:37 dist-test-slave-98mk krb5kdc[20968](info): commencing operation
krb5kdc: starting...
W20260929 22:50:38.324278 14927 mini_kdc.cc:121] Time spent starting KDC: real 0.939s user 0.001s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790722213368738-14927-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790722213368738-14927-0/krb5kdc/test-user.keytab.
Sep 29 22:50:38 dist-test-slave-98mk krb5kdc[20968](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790722238, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T22:50:38Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T22:50:38Z Disabled control of system clock
WARNING: no policy specified for oryx/127.14.147.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.14.147.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.14.147.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790722213368738-14927-0/krb5kdc/oryx_127.14.147.254.keytab.
Entry for principal oryx/127.14.147.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790722213368738-14927-0/krb5kdc/oryx_127.14.147.254.keytab.
I20260929 22:50:38.441781 14927 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhWOCLk/build/release/bin/kudu
/tmp/dist-test-taskhWOCLk/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790722213368738-14927-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790722213368738-14927-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790722213368738-14927-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790722213368738-14927-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.147.254:45601
--webserver_interface=127.14.147.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790722213368738-14927-0/krb5kdc/oryx_127.14.147.254.keytab
--principal=oryx/127.14.147.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.147.212:34343
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.147.254:45601
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790722213368738-14927-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790722213368738-14927-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790722213368738-14927-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 22:50:38.521570 20982 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 22:50:38.521793 20982 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 22:50:38.521816 20982 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 22:50:38.523011 20982 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 22:50:38.523059 20982 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 22:50:38.523072 20982 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 22:50:38.523083 20982 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 22:50:38.523096 20982 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 22:50:38.524678 20982 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.147.212:34343
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790722213368738-14927-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790722213368738-14927-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.14.147.254:45601
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790722213368738-14927-0/krb5kdc/oryx_127.14.147.254.keytab
--principal=oryx/127.14.147.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.147.254:45601
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790722213368738-14927-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.14.147.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20982
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790722213368738-14927-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 21:58:54 UTC on ccycloud.quasar-xguoti.root.comops.site
build id 4681390
I20260929 22:50:38.524932 20982 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 22:50:38.525146 20982 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 22:50:38.529740 20990 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 22:50:38.529843 20988 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 22:50:38.529775 20987 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 22:50:38.530190 20982 server_base.cc:1034] running on GCE node
I20260929 22:50:38.530432 20982 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 22:50:38.530726 20982 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 22:50:38.531911 20982 hybrid_clock.cc:648] HybridClock initialized: now 1790722238531893 us; error 28 us; skew 500 ppm
Sep 29 22:50:38 dist-test-slave-98mk krb5kdc[20968](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790722238, etypes {rep=17 tkt=17 ses=17}, oryx/127.14.147.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 22:50:38.534264 20982 init.cc:377] Logged in from keytab as oryx/127.14.147.254@KRBTEST.COM (short username oryx)
I20260929 22:50:38.534672 20982 webserver.cc:460] Webserver started at http://127.14.147.254:44995/ using document root <none> and password file <none>
I20260929 22:50:38.534878 20982 fs_manager.cc:359] Metadata directory not provided
I20260929 22:50:38.534924 20982 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 22:50:38.535044 20982 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 22:50:38.536147 20982 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790722213368738-14927-0/minicluster-data/master-0/data/instance:
uuid: "d38d0f6f0d3c4d2b8b7131e3f0f72ffe"
format_stamp: "Formatted at 2026-09-29 22:50:38 on dist-test-slave-98mk"
server_key: "6aee63c587f96f578b07210e1a2c453d"
server_key_iv: "97323f3904ab98bb91c8a2ecdc413f82"
server_key_version: "encryptionkey@0"
I20260929 22:50:38.536473 20982 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790722213368738-14927-0/minicluster-data/master-0/wal/instance:
uuid: "d38d0f6f0d3c4d2b8b7131e3f0f72ffe"
format_stamp: "Formatted at 2026-09-29 22:50:38 on dist-test-slave-98mk"
server_key: "6aee63c587f96f578b07210e1a2c453d"
server_key_iv: "97323f3904ab98bb91c8a2ecdc413f82"
server_key_version: "encryptionkey@0"
I20260929 22:50:38.537726 20982 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260929 22:50:38.538460 20997 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 22:50:38.538627 20982 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260929 22:50:38.538709 20982 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790722213368738-14927-0/minicluster-data/master-0/data,/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790722213368738-14927-0/minicluster-data/master-0/wal
uuid: "d38d0f6f0d3c4d2b8b7131e3f0f72ffe"
format_stamp: "Formatted at 2026-09-29 22:50:38 on dist-test-slave-98mk"
server_key: "6aee63c587f96f578b07210e1a2c453d"
server_key_iv: "97323f3904ab98bb91c8a2ecdc413f82"
server_key_version: "encryptionkey@0"
I20260929 22:50:38.538769 20982 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790722213368738-14927-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790722213368738-14927-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790722213368738-14927-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 22:50:38.561832 20982 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 22:50:38.563611 20982 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 22:50:38.563747 20982 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 22:50:38.567317 20982 rpc_server.cc:307] RPC server started. Bound to: 127.14.147.254:45601
I20260929 22:50:38.567358 21049 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.147.254:45601 every 8 connection(s)
I20260929 22:50:38.567689 20982 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790722213368738-14927-0/minicluster-data/master-0/data/info.pb
I20260929 22:50:38.568585 21050 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 22:50:38.570827 21050 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d38d0f6f0d3c4d2b8b7131e3f0f72ffe: Bootstrap starting.
I20260929 22:50:38.571357 21050 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d38d0f6f0d3c4d2b8b7131e3f0f72ffe: Neither blocks nor log segments found. Creating new log.
I20260929 22:50:38.571635 21050 log.cc:824] T 00000000000000000000000000000000 P d38d0f6f0d3c4d2b8b7131e3f0f72ffe: Log is configured to *not* fsync() on all Append() calls
I20260929 22:50:38.572328 21050 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d38d0f6f0d3c4d2b8b7131e3f0f72ffe: No bootstrap required, opened a new log
I20260929 22:50:38.573851 21050 raft_consensus.cc:348] T 00000000000000000000000000000000 P d38d0f6f0d3c4d2b8b7131e3f0f72ffe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d38d0f6f0d3c4d2b8b7131e3f0f72ffe" member_type: VOTER last_known_addr { host: "127.14.147.254" port: 45601 } }
I20260929 22:50:38.573992 21050 raft_consensus.cc:374] T 00000000000000000000000000000000 P d38d0f6f0d3c4d2b8b7131e3f0f72ffe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 22:50:38.574011 21050 raft_consensus.cc:729] T 00000000000000000000000000000000 P d38d0f6f0d3c4d2b8b7131e3f0f72ffe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d38d0f6f0d3c4d2b8b7131e3f0f72ffe, State: Initialized, Role: FOLLOWER
I20260929 22:50:38.574154 21050 consensus_queue.cc:260] T 00000000000000000000000000000000 P d38d0f6f0d3c4d2b8b7131e3f0f72ffe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d38d0f6f0d3c4d2b8b7131e3f0f72ffe" member_type: VOTER last_known_addr { host: "127.14.147.254" port: 45601 } }
I20260929 22:50:38.574215 21050 raft_consensus.cc:388] T 00000000000000000000000000000000 P d38d0f6f0d3c4d2b8b7131e3f0f72ffe [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 22:50:38.574236 21050 raft_consensus.cc:482] T 00000000000000000000000000000000 P d38d0f6f0d3c4d2b8b7131e3f0f72ffe [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 22:50:38.574280 21050 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d38d0f6f0d3c4d2b8b7131e3f0f72ffe [term 0 FOLLOWER]: Advancing to term 1
I20260929 22:50:38.575028 21050 raft_consensus.cc:504] T 00000000000000000000000000000000 P d38d0f6f0d3c4d2b8b7131e3f0f72ffe [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d38d0f6f0d3c4d2b8b7131e3f0f72ffe" member_type: VOTER last_known_addr { host: "127.14.147.254" port: 45601 } }
I20260929 22:50:38.575116 21050 leader_election.cc:302] T 00000000000000000000000000000000 P d38d0f6f0d3c4d2b8b7131e3f0f72ffe [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d38d0f6f0d3c4d2b8b7131e3f0f72ffe; no voters:
I20260929 22:50:38.575268 21050 leader_election.cc:288] T 00000000000000000000000000000000 P d38d0f6f0d3c4d2b8b7131e3f0f72ffe [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 22:50:38.575297 21053 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d38d0f6f0d3c4d2b8b7131e3f0f72ffe [term 1 FOLLOWER]: Leader election won for term 1
I20260929 22:50:38.575440 21053 raft_consensus.cc:686] T 00000000000000000000000000000000 P d38d0f6f0d3c4d2b8b7131e3f0f72ffe [term 1 LEADER]: Becoming Leader. State: Replica: d38d0f6f0d3c4d2b8b7131e3f0f72ffe, State: Running, Role: LEADER
I20260929 22:50:38.575521 21050 sys_catalog.cc:564] T 00000000000000000000000000000000 P d38d0f6f0d3c4d2b8b7131e3f0f72ffe [sys.catalog]: configured and running, proceeding with master startup.
I20260929 22:50:38.575529 21053 consensus_queue.cc:237] T 00000000000000000000000000000000 P d38d0f6f0d3c4d2b8b7131e3f0f72ffe [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d38d0f6f0d3c4d2b8b7131e3f0f72ffe" member_type: VOTER last_known_addr { host: "127.14.147.254" port: 45601 } }
I20260929 22:50:38.575871 21054 sys_catalog.cc:455] T 00000000000000000000000000000000 P d38d0f6f0d3c4d2b8b7131e3f0f72ffe [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d38d0f6f0d3c4d2b8b7131e3f0f72ffe" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d38d0f6f0d3c4d2b8b7131e3f0f72ffe" member_type: VOTER last_known_addr { host: "127.14.147.254" port: 45601 } } }
I20260929 22:50:38.575959 21054 sys_catalog.cc:458] T 00000000000000000000000000000000 P d38d0f6f0d3c4d2b8b7131e3f0f72ffe [sys.catalog]: This master's current role is: LEADER
I20260929 22:50:38.575881 21055 sys_catalog.cc:455] T 00000000000000000000000000000000 P d38d0f6f0d3c4d2b8b7131e3f0f72ffe [sys.catalog]: SysCatalogTable state changed. Reason: New leader d38d0f6f0d3c4d2b8b7131e3f0f72ffe. Latest consensus state: current_term: 1 leader_uuid: "d38d0f6f0d3c4d2b8b7131e3f0f72ffe" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d38d0f6f0d3c4d2b8b7131e3f0f72ffe" member_type: VOTER last_known_addr { host: "127.14.147.254" port: 45601 } } }
I20260929 22:50:38.576092 21055 sys_catalog.cc:458] T 00000000000000000000000000000000 P d38d0f6f0d3c4d2b8b7131e3f0f72ffe [sys.catalog]: This master's current role is: LEADER
I20260929 22:50:38.576419 21060 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 22:50:38.576785 14927 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhWOCLk/build/release/bin/kudu as pid 20982
I20260929 22:50:38.576893 14927 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790722213368738-14927-0/minicluster-data/master-0/wal/instance
I20260929 22:50:38.576890 21060 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 22:50:38.577095 14927 external_mini_cluster.cc:1428] Setting key 40c449efadd3457da12d0b2430066f17
I20260929 22:50:38.579787 21060 catalog_manager.cc:1347] Generated new cluster ID: eeb78789c1ae4d64a674c24edf4ad136
I20260929 22:50:38.579838 21060 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 29 22:50:38 dist-test-slave-98mk krb5kdc[20968](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790722238, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.14.147.254@KRBTEST.COM
I20260929 22:50:38.585858 21060 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 22:50:38.586522 21060 catalog_manager.cc:1504] Loading token signing keys...
I20260929 22:50:38.588230 21073 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:50:38.577558 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53102 (local address 127.14.147.254:45601)
0929 22:50:38.577704 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0929 22:50:38.577713 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 22:50:38.577744 (+ 31us) server_negotiation.cc:407] Connection header received
0929 22:50:38.577851 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 22:50:38.577870 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 22:50:38.577926 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 22:50:38.578061 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0929 22:50:38.578620 (+ 559us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:50:38.580083 (+ 1463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:50:38.581109 (+ 1026us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:50:38.581742 (+ 633us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:50:38.581884 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 22:50:38.584562 (+ 2678us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 22:50:38.584567 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 22:50:38.584567 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 22:50:38.584580 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0929 22:50:38.586625 (+ 2045us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 22:50:38.587034 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 22:50:38.587037 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 22:50:38.587038 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 22:50:38.587139 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 22:50:38.587320 (+ 181us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 22:50:38.587320 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 22:50:38.587321 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 22:50:38.587609 (+ 288us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 22:50:38.587760 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0929 22:50:38.587868 (+ 108us) server_negotiation.cc:299] Negotiation successful
0929 22:50:38.588008 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":101,"thread_start_us":76,"threads_started":1}
I20260929 22:50:38.590623 21060 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d38d0f6f0d3c4d2b8b7131e3f0f72ffe: Generated new TSK 0
I20260929 22:50:38.590775 21060 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.14.147.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.14.147.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.14.147.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790722213368738-14927-0/krb5kdc/oryx_127.14.147.193.keytab.
Entry for principal oryx/127.14.147.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790722213368738-14927-0/krb5kdc/oryx_127.14.147.193.keytab.
I20260929 22:50:38.615643 14927 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhWOCLk/build/release/bin/kudu
/tmp/dist-test-taskhWOCLk/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790722213368738-14927-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790722213368738-14927-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790722213368738-14927-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790722213368738-14927-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.147.193:0
--local_ip_for_outbound_sockets=127.14.147.193
--webserver_interface=127.14.147.193
--webserver_port=0
--tserver_master_addrs=127.14.147.254:45601
--keytab_file=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790722213368738-14927-0/krb5kdc/oryx_127.14.147.193.keytab
--principal=oryx/127.14.147.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.147.212:34343
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790722213368738-14927-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790722213368738-14927-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790722213368738-14927-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 22:50:38.696944 21077 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 22:50:38.697175 21077 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 22:50:38.697203 21077 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 22:50:38.698487 21077 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 22:50:38.698539 21077 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 22:50:38.698568 21077 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.147.193
I20260929 22:50:38.700255 21077 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.147.212:34343
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790722213368738-14927-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790722213368738-14927-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790722213368738-14927-0/krb5kdc/oryx_127.14.147.193.keytab
--principal=oryx/127.14.147.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.147.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790722213368738-14927-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.14.147.193
--webserver_port=0
--tserver_master_addrs=127.14.147.254:45601
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21077
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.147.193
--log_dir=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790722213368738-14927-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 21:58:54 UTC on ccycloud.quasar-xguoti.root.comops.site
build id 4681390
I20260929 22:50:38.700510 21077 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 22:50:38.700747 21077 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 22:50:38.705948 21082 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 22:50:38.705952 21085 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 22:50:38.706058 21077 server_base.cc:1034] running on GCE node
W20260929 22:50:38.705952 21083 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 22:50:38.706355 21077 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 22:50:38.706591 21077 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 22:50:38.707741 21077 hybrid_clock.cc:648] HybridClock initialized: now 1790722238707724 us; error 36 us; skew 500 ppm
Sep 29 22:50:38 dist-test-slave-98mk krb5kdc[20968](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790722238, etypes {rep=17 tkt=17 ses=17}, oryx/127.14.147.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 22:50:38.711032 21077 init.cc:377] Logged in from keytab as oryx/127.14.147.193@KRBTEST.COM (short username oryx)
I20260929 22:50:38.711563 21077 webserver.cc:460] Webserver started at http://127.14.147.193:41813/ using document root <none> and password file <none>
I20260929 22:50:38.711789 21077 fs_manager.cc:359] Metadata directory not provided
I20260929 22:50:38.711846 21077 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 22:50:38.711970 21077 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 22:50:38.713017 21077 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790722213368738-14927-0/minicluster-data/ts-0/data/instance:
uuid: "e0d74c61684f40338bc384332f90e9c7"
format_stamp: "Formatted at 2026-09-29 22:50:38 on dist-test-slave-98mk"
server_key: "478a426444f2df994ac591f1b7a8a906"
server_key_iv: "87a860fe59b1ad8c2d4e452a67005836"
server_key_version: "encryptionkey@0"
I20260929 22:50:38.713346 21077 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790722213368738-14927-0/minicluster-data/ts-0/wal/instance:
uuid: "e0d74c61684f40338bc384332f90e9c7"
format_stamp: "Formatted at 2026-09-29 22:50:38 on dist-test-slave-98mk"
server_key: "478a426444f2df994ac591f1b7a8a906"
server_key_iv: "87a860fe59b1ad8c2d4e452a67005836"
server_key_version: "encryptionkey@0"
I20260929 22:50:38.714592 21077 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260929 22:50:38.715317 21092 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 22:50:38.715523 21077 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260929 22:50:38.715612 21077 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790722213368738-14927-0/minicluster-data/ts-0/data,/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790722213368738-14927-0/minicluster-data/ts-0/wal
uuid: "e0d74c61684f40338bc384332f90e9c7"
format_stamp: "Formatted at 2026-09-29 22:50:38 on dist-test-slave-98mk"
server_key: "478a426444f2df994ac591f1b7a8a906"
server_key_iv: "87a860fe59b1ad8c2d4e452a67005836"
server_key_version: "encryptionkey@0"
I20260929 22:50:38.715677 21077 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790722213368738-14927-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790722213368738-14927-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790722213368738-14927-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 22:50:38.728706 21077 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 22:50:38.731093 21077 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 22:50:38.731259 21077 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 22:50:38.731520 21077 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 22:50:38.731835 21077 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 22:50:38.731870 21077 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 22:50:38.731900 21077 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 22:50:38.731926 21077 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 22:50:38.737794 21077 rpc_server.cc:307] RPC server started. Bound to: 127.14.147.193:44967
I20260929 22:50:38.737857 21206 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.147.193:44967 every 8 connection(s)
I20260929 22:50:38.738161 21077 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790722213368738-14927-0/minicluster-data/ts-0/data/info.pb
I20260929 22:50:38.740288 14927 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhWOCLk/build/release/bin/kudu as pid 21077
I20260929 22:50:38.740399 14927 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790722213368738-14927-0/minicluster-data/ts-0/wal/instance
I20260929 22:50:38.740599 14927 external_mini_cluster.cc:1428] Setting key 6da0684e6ed8f5b360efbbdb9d82832c
Sep 29 22:50:38 dist-test-slave-98mk krb5kdc[20968](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790722238, etypes {rep=17 tkt=17 ses=17}, oryx/127.14.147.193@KRBTEST.COM for oryx/127.14.147.254@KRBTEST.COM
I20260929 22:50:38.748835 21073 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:50:38.739002 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.147.193:55049 (local address 127.14.147.254:45601)
0929 22:50:38.739089 (+ 87us) server_negotiation.cc:206] Beginning negotiation
0929 22:50:38.739091 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 22:50:38.739311 (+ 220us) server_negotiation.cc:407] Connection header received
0929 22:50:38.739801 (+ 490us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 22:50:38.739802 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 22:50:38.739813 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 22:50:38.739872 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0929 22:50:38.741037 (+ 1165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:50:38.741650 (+ 613us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:50:38.743047 (+ 1397us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:50:38.743276 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:50:38.743335 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 22:50:38.745925 (+ 2590us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 22:50:38.745926 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 22:50:38.745926 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 22:50:38.745934 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0929 22:50:38.747221 (+ 1287us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 22:50:38.747741 (+ 520us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 22:50:38.747743 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 22:50:38.747743 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 22:50:38.747804 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 22:50:38.748104 (+ 300us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 22:50:38.748105 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 22:50:38.748105 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 22:50:38.748225 (+ 120us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 22:50:38.748287 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0929 22:50:38.748672 (+ 385us) server_negotiation.cc:299] Negotiation successful
0929 22:50:38.748759 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
WARNING: no policy specified for oryx/127.14.147.194@KRBTEST.COM; defaulting to no policy
I20260929 22:50:38.749022 21209 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:50:38.738940 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.147.254:45601 (local address 127.14.147.193:55049)
0929 22:50:38.739162 (+ 222us) negotiation.cc:107] Waiting for socket to connect
0929 22:50:38.739202 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0929 22:50:38.739717 (+ 515us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 22:50:38.739938 (+ 221us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 22:50:38.739942 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 22:50:38.740677 (+ 735us) client_negotiation.cc:189] Negotiated authn=SASL
0929 22:50:38.740949 (+ 272us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 22:50:38.740956 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 22:50:38.741736 (+ 780us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 22:50:38.741737 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 22:50:38.742961 (+ 1224us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 22:50:38.742966 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 22:50:38.743348 (+ 382us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 22:50:38.743349 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 22:50:38.743513 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 22:50:38.744014 (+ 501us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 22:50:38.744023 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0929 22:50:38.745795 (+ 1772us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 22:50:38.747313 (+ 1518us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 22:50:38.747322 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 22:50:38.747324 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 22:50:38.747663 (+ 339us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 22:50:38.747874 (+ 211us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 22:50:38.747875 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 22:50:38.747876 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 22:50:38.748023 (+ 147us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 22:50:38.748293 (+ 270us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 22:50:38.748301 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 22:50:38.748487 (+ 186us) client_negotiation.cc:769] Sending connection context
0929 22:50:38.748604 (+ 117us) client_negotiation.cc:240] Negotiation successful
0929 22:50:38.748781 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":184,"thread_start_us":72,"threads_started":1}
I20260929 22:50:38.749497 21207 heartbeater.cc:347] Connected to a master server at 127.14.147.254:45601
I20260929 22:50:38.749596 21207 heartbeater.cc:464] Registering TS with master...
I20260929 22:50:38.749776 21207 heartbeater.cc:511] Master 127.14.147.254:45601 requested a full tablet report, sending...
I20260929 22:50:38.750278 21014 ts_manager.cc:194] Registered new tserver with Master: e0d74c61684f40338bc384332f90e9c7 (127.14.147.193:44967)
I20260929 22:50:38.750901 21014 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.14.147.193@KRBTEST.COM'} at 127.14.147.193:55049
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.14.147.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.14.147.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790722213368738-14927-0/krb5kdc/oryx_127.14.147.194.keytab.
Entry for principal oryx/127.14.147.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790722213368738-14927-0/krb5kdc/oryx_127.14.147.194.keytab.
I20260929 22:50:38.765524 14927 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhWOCLk/build/release/bin/kudu
/tmp/dist-test-taskhWOCLk/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790722213368738-14927-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790722213368738-14927-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790722213368738-14927-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790722213368738-14927-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.147.194:0
--local_ip_for_outbound_sockets=127.14.147.194
--webserver_interface=127.14.147.194
--webserver_port=0
--tserver_master_addrs=127.14.147.254:45601
--keytab_file=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790722213368738-14927-0/krb5kdc/oryx_127.14.147.194.keytab
--principal=oryx/127.14.147.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.147.212:34343
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790722213368738-14927-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790722213368738-14927-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790722213368738-14927-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 22:50:38.846624 21212 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 22:50:38.846865 21212 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 22:50:38.846901 21212 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 22:50:38.848295 21212 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 22:50:38.848351 21212 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 22:50:38.848390 21212 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.147.194
I20260929 22:50:38.850126 21212 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.147.212:34343
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790722213368738-14927-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790722213368738-14927-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790722213368738-14927-0/krb5kdc/oryx_127.14.147.194.keytab
--principal=oryx/127.14.147.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.147.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790722213368738-14927-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.14.147.194
--webserver_port=0
--tserver_master_addrs=127.14.147.254:45601
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21212
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.147.194
--log_dir=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790722213368738-14927-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 21:58:54 UTC on ccycloud.quasar-xguoti.root.comops.site
build id 4681390
I20260929 22:50:38.850414 21212 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 22:50:38.850634 21212 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 22:50:38.855317 21217 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 22:50:38.855317 21220 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 22:50:38.855578 21212 server_base.cc:1034] running on GCE node
W20260929 22:50:38.855367 21218 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 22:50:38.855800 21212 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 22:50:38.856042 21212 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 22:50:38.857203 21212 hybrid_clock.cc:648] HybridClock initialized: now 1790722238857172 us; error 45 us; skew 500 ppm
Sep 29 22:50:38 dist-test-slave-98mk krb5kdc[20968](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790722238, etypes {rep=17 tkt=17 ses=17}, oryx/127.14.147.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 22:50:38.859716 21212 init.cc:377] Logged in from keytab as oryx/127.14.147.194@KRBTEST.COM (short username oryx)
I20260929 22:50:38.860144 21212 webserver.cc:460] Webserver started at http://127.14.147.194:42563/ using document root <none> and password file <none>
I20260929 22:50:38.860357 21212 fs_manager.cc:359] Metadata directory not provided
I20260929 22:50:38.860401 21212 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 22:50:38.860522 21212 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 22:50:38.861541 21212 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790722213368738-14927-0/minicluster-data/ts-1/data/instance:
uuid: "ab4a15db2c3f422e8646d4b5255cf8df"
format_stamp: "Formatted at 2026-09-29 22:50:38 on dist-test-slave-98mk"
server_key: "1efd4d2d11fa0ea992729dde3a4bc74b"
server_key_iv: "f8771db19f7a73734110f71bc7974c91"
server_key_version: "encryptionkey@0"
I20260929 22:50:38.861858 21212 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790722213368738-14927-0/minicluster-data/ts-1/wal/instance:
uuid: "ab4a15db2c3f422e8646d4b5255cf8df"
format_stamp: "Formatted at 2026-09-29 22:50:38 on dist-test-slave-98mk"
server_key: "1efd4d2d11fa0ea992729dde3a4bc74b"
server_key_iv: "f8771db19f7a73734110f71bc7974c91"
server_key_version: "encryptionkey@0"
I20260929 22:50:38.863107 21212 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260929 22:50:38.863824 21227 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 22:50:38.863972 21212 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260929 22:50:38.864055 21212 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790722213368738-14927-0/minicluster-data/ts-1/data,/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790722213368738-14927-0/minicluster-data/ts-1/wal
uuid: "ab4a15db2c3f422e8646d4b5255cf8df"
format_stamp: "Formatted at 2026-09-29 22:50:38 on dist-test-slave-98mk"
server_key: "1efd4d2d11fa0ea992729dde3a4bc74b"
server_key_iv: "f8771db19f7a73734110f71bc7974c91"
server_key_version: "encryptionkey@0"
I20260929 22:50:38.864125 21212 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790722213368738-14927-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790722213368738-14927-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790722213368738-14927-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 22:50:38.875854 21212 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 22:50:38.877815 21212 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 22:50:38.877949 21212 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 22:50:38.878170 21212 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 22:50:38.878459 21212 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 22:50:38.878491 21212 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 22:50:38.878513 21212 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 22:50:38.878535 21212 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 22:50:38.883896 21212 rpc_server.cc:307] RPC server started. Bound to: 127.14.147.194:45475
I20260929 22:50:38.883960 21341 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.147.194:45475 every 8 connection(s)
I20260929 22:50:38.884258 21212 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790722213368738-14927-0/minicluster-data/ts-1/data/info.pb
I20260929 22:50:38.890117 14927 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhWOCLk/build/release/bin/kudu as pid 21212
I20260929 22:50:38.890221 14927 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790722213368738-14927-0/minicluster-data/ts-1/wal/instance
Sep 29 22:50:38 dist-test-slave-98mk krb5kdc[20968](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790722238, etypes {rep=17 tkt=17 ses=17}, oryx/127.14.147.194@KRBTEST.COM for oryx/127.14.147.254@KRBTEST.COM
I20260929 22:50:38.890401 14927 external_mini_cluster.cc:1428] Setting key 34d767073bd02483b858b7f41061ed61
I20260929 22:50:38.893422 21073 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:50:38.884931 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.147.194:33497 (local address 127.14.147.254:45601)
0929 22:50:38.884972 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0929 22:50:38.884974 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 22:50:38.885177 (+ 203us) server_negotiation.cc:407] Connection header received
0929 22:50:38.885709 (+ 532us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 22:50:38.885710 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 22:50:38.885722 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 22:50:38.885774 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0929 22:50:38.886741 (+ 967us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:50:38.887235 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:50:38.888259 (+ 1024us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:50:38.888449 (+ 190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:50:38.888506 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 22:50:38.890736 (+ 2230us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 22:50:38.890737 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 22:50:38.890738 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 22:50:38.890744 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0929 22:50:38.892106 (+ 1362us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 22:50:38.892495 (+ 389us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 22:50:38.892496 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 22:50:38.892497 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 22:50:38.892557 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 22:50:38.892795 (+ 238us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 22:50:38.892796 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 22:50:38.892797 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 22:50:38.892914 (+ 117us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 22:50:38.892969 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0929 22:50:38.893246 (+ 277us) server_negotiation.cc:299] Negotiation successful
0929 22:50:38.893329 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260929 22:50:38.893657 21344 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:50:38.884905 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.147.254:45601 (local address 127.14.147.194:33497)
0929 22:50:38.885063 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0929 22:50:38.885100 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0929 22:50:38.885609 (+ 509us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 22:50:38.885872 (+ 263us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 22:50:38.885878 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 22:50:38.886453 (+ 575us) client_negotiation.cc:189] Negotiated authn=SASL
0929 22:50:38.886646 (+ 193us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 22:50:38.886650 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 22:50:38.887328 (+ 678us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 22:50:38.887329 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 22:50:38.888175 (+ 846us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 22:50:38.888178 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 22:50:38.888523 (+ 345us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 22:50:38.888524 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 22:50:38.888659 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 22:50:38.889201 (+ 542us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 22:50:38.889211 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0929 22:50:38.890653 (+ 1442us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 22:50:38.892183 (+ 1530us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 22:50:38.892191 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 22:50:38.892194 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 22:50:38.892426 (+ 232us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 22:50:38.892612 (+ 186us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 22:50:38.892613 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 22:50:38.892613 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 22:50:38.892738 (+ 125us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 22:50:38.892968 (+ 230us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 22:50:38.892975 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 22:50:38.893118 (+ 143us) client_negotiation.cc:769] Sending connection context
0929 22:50:38.893218 (+ 100us) client_negotiation.cc:240] Negotiation successful
0929 22:50:38.893388 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":76,"threads_started":1}
I20260929 22:50:38.894196 21342 heartbeater.cc:347] Connected to a master server at 127.14.147.254:45601
I20260929 22:50:38.894317 21342 heartbeater.cc:464] Registering TS with master...
I20260929 22:50:38.894536 21342 heartbeater.cc:511] Master 127.14.147.254:45601 requested a full tablet report, sending...
I20260929 22:50:38.894977 21014 ts_manager.cc:194] Registered new tserver with Master: ab4a15db2c3f422e8646d4b5255cf8df (127.14.147.194:45475)
I20260929 22:50:38.895354 21014 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.14.147.194@KRBTEST.COM'} at 127.14.147.194:33497
WARNING: no policy specified for oryx/127.14.147.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.14.147.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.14.147.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790722213368738-14927-0/krb5kdc/oryx_127.14.147.195.keytab.
Entry for principal oryx/127.14.147.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790722213368738-14927-0/krb5kdc/oryx_127.14.147.195.keytab.
I20260929 22:50:38.914423 14927 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhWOCLk/build/release/bin/kudu
/tmp/dist-test-taskhWOCLk/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790722213368738-14927-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790722213368738-14927-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790722213368738-14927-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790722213368738-14927-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.147.195:0
--local_ip_for_outbound_sockets=127.14.147.195
--webserver_interface=127.14.147.195
--webserver_port=0
--tserver_master_addrs=127.14.147.254:45601
--keytab_file=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790722213368738-14927-0/krb5kdc/oryx_127.14.147.195.keytab
--principal=oryx/127.14.147.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.147.212:34343
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790722213368738-14927-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790722213368738-14927-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790722213368738-14927-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 22:50:38.998445 21347 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 22:50:38.998682 21347 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 22:50:38.998718 21347 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 22:50:38.999980 21347 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 22:50:39.000028 21347 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 22:50:39.000067 21347 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.147.195
I20260929 22:50:39.001643 21347 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.147.212:34343
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790722213368738-14927-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790722213368738-14927-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790722213368738-14927-0/krb5kdc/oryx_127.14.147.195.keytab
--principal=oryx/127.14.147.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.147.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790722213368738-14927-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.14.147.195
--webserver_port=0
--tserver_master_addrs=127.14.147.254:45601
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21347
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.147.195
--log_dir=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790722213368738-14927-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 21:58:54 UTC on ccycloud.quasar-xguoti.root.comops.site
build id 4681390
I20260929 22:50:39.001911 21347 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 22:50:39.002132 21347 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 22:50:39.006887 21355 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 22:50:39.006925 21347 server_base.cc:1034] running on GCE node
W20260929 22:50:39.006892 21352 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 22:50:39.006892 21353 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 22:50:39.007287 21347 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 22:50:39.007555 21347 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 22:50:39.008721 21347 hybrid_clock.cc:648] HybridClock initialized: now 1790722239008704 us; error 27 us; skew 500 ppm
Sep 29 22:50:39 dist-test-slave-98mk krb5kdc[20968](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790722239, etypes {rep=17 tkt=17 ses=17}, oryx/127.14.147.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 22:50:39.011332 21347 init.cc:377] Logged in from keytab as oryx/127.14.147.195@KRBTEST.COM (short username oryx)
I20260929 22:50:39.011771 21347 webserver.cc:460] Webserver started at http://127.14.147.195:45097/ using document root <none> and password file <none>
I20260929 22:50:39.011983 21347 fs_manager.cc:359] Metadata directory not provided
I20260929 22:50:39.012032 21347 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 22:50:39.012156 21347 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 22:50:39.013187 21347 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790722213368738-14927-0/minicluster-data/ts-2/data/instance:
uuid: "f9ed52b3ef574d3d8c17014dbf842a54"
format_stamp: "Formatted at 2026-09-29 22:50:39 on dist-test-slave-98mk"
server_key: "b96fa853a9eaa6a848b9328200c3a5fd"
server_key_iv: "5bfb613a395ad47c0a7386f0eadced5a"
server_key_version: "encryptionkey@0"
I20260929 22:50:39.013511 21347 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790722213368738-14927-0/minicluster-data/ts-2/wal/instance:
uuid: "f9ed52b3ef574d3d8c17014dbf842a54"
format_stamp: "Formatted at 2026-09-29 22:50:39 on dist-test-slave-98mk"
server_key: "b96fa853a9eaa6a848b9328200c3a5fd"
server_key_iv: "5bfb613a395ad47c0a7386f0eadced5a"
server_key_version: "encryptionkey@0"
I20260929 22:50:39.014819 21347 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260929 22:50:39.015522 21362 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 22:50:39.015661 21347 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260929 22:50:39.015743 21347 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790722213368738-14927-0/minicluster-data/ts-2/data,/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790722213368738-14927-0/minicluster-data/ts-2/wal
uuid: "f9ed52b3ef574d3d8c17014dbf842a54"
format_stamp: "Formatted at 2026-09-29 22:50:39 on dist-test-slave-98mk"
server_key: "b96fa853a9eaa6a848b9328200c3a5fd"
server_key_iv: "5bfb613a395ad47c0a7386f0eadced5a"
server_key_version: "encryptionkey@0"
I20260929 22:50:39.015810 21347 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790722213368738-14927-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790722213368738-14927-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790722213368738-14927-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 22:50:39.030961 21347 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 22:50:39.033078 21347 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 22:50:39.033218 21347 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 22:50:39.033438 21347 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 22:50:39.033739 21347 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 22:50:39.033777 21347 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 22:50:39.033807 21347 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 22:50:39.033829 21347 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 22:50:39.039168 21347 rpc_server.cc:307] RPC server started. Bound to: 127.14.147.195:38905
I20260929 22:50:39.039222 21475 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.147.195:38905 every 8 connection(s)
I20260929 22:50:39.039561 21347 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790722213368738-14927-0/minicluster-data/ts-2/data/info.pb
Sep 29 22:50:39 dist-test-slave-98mk krb5kdc[20968](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790722239, etypes {rep=17 tkt=17 ses=17}, oryx/127.14.147.195@KRBTEST.COM for oryx/127.14.147.254@KRBTEST.COM
I20260929 22:50:39.048271 21073 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:50:39.040264 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.147.195:46551 (local address 127.14.147.254:45601)
0929 22:50:39.040314 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0929 22:50:39.040316 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 22:50:39.040518 (+ 202us) server_negotiation.cc:407] Connection header received
0929 22:50:39.041049 (+ 531us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 22:50:39.041051 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 22:50:39.041063 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 22:50:39.041120 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0929 22:50:39.042109 (+ 989us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:50:39.042623 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:50:39.043625 (+ 1002us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:50:39.043813 (+ 188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:50:39.043868 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 22:50:39.045869 (+ 2001us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 22:50:39.045870 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 22:50:39.045870 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 22:50:39.045877 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0929 22:50:39.046982 (+ 1105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 22:50:39.047372 (+ 390us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 22:50:39.047374 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 22:50:39.047374 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 22:50:39.047458 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 22:50:39.047672 (+ 214us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 22:50:39.047673 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 22:50:39.047674 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 22:50:39.047784 (+ 110us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 22:50:39.047838 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0929 22:50:39.048091 (+ 253us) server_negotiation.cc:299] Negotiation successful
0929 22:50:39.048169 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260929 22:50:39.048427 21478 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:50:39.040190 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.147.254:45601 (local address 127.14.147.195:46551)
0929 22:50:39.040396 (+ 206us) negotiation.cc:107] Waiting for socket to connect
0929 22:50:39.040433 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0929 22:50:39.040954 (+ 521us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 22:50:39.041192 (+ 238us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 22:50:39.041197 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 22:50:39.041814 (+ 617us) client_negotiation.cc:189] Negotiated authn=SASL
0929 22:50:39.042022 (+ 208us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 22:50:39.042027 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 22:50:39.042713 (+ 686us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 22:50:39.042714 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 22:50:39.043555 (+ 841us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 22:50:39.043557 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 22:50:39.043881 (+ 324us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 22:50:39.043882 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 22:50:39.044003 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 22:50:39.044477 (+ 474us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 22:50:39.044486 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0929 22:50:39.045768 (+ 1282us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 22:50:39.047075 (+ 1307us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 22:50:39.047082 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 22:50:39.047084 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 22:50:39.047303 (+ 219us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 22:50:39.047513 (+ 210us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 22:50:39.047513 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 22:50:39.047514 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 22:50:39.047615 (+ 101us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 22:50:39.047838 (+ 223us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 22:50:39.047845 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 22:50:39.047976 (+ 131us) client_negotiation.cc:769] Sending connection context
0929 22:50:39.048062 (+ 86us) client_negotiation.cc:240] Negotiation successful
0929 22:50:39.048217 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":162,"thread_start_us":98,"threads_started":1}
I20260929 22:50:39.048846 21476 heartbeater.cc:347] Connected to a master server at 127.14.147.254:45601
I20260929 22:50:39.048946 21476 heartbeater.cc:464] Registering TS with master...
I20260929 22:50:39.049108 21476 heartbeater.cc:511] Master 127.14.147.254:45601 requested a full tablet report, sending...
I20260929 22:50:39.049398 14927 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhWOCLk/build/release/bin/kudu as pid 21347
I20260929 22:50:39.049501 14927 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790722213368738-14927-0/minicluster-data/ts-2/wal/instance
I20260929 22:50:39.049535 21014 ts_manager.cc:194] Registered new tserver with Master: f9ed52b3ef574d3d8c17014dbf842a54 (127.14.147.195:38905)
I20260929 22:50:39.049685 14927 external_mini_cluster.cc:1428] Setting key 9345827983c08c82629318a82ae98fd7
I20260929 22:50:39.049958 21014 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.14.147.195@KRBTEST.COM'} at 127.14.147.195:46551
I20260929 22:50:39.050503 14927 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.14.147.253@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.14.147.253@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.14.147.253 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790722213368738-14927-0/krb5kdc/oryx_127.14.147.253.keytab.
Entry for principal oryx/127.14.147.253 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790722213368738-14927-0/krb5kdc/oryx_127.14.147.253.keytab.
I20260929 22:50:39.073498 14927 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhWOCLk/build/release/bin/kudu
/tmp/dist-test-taskhWOCLk/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790722213368738-14927-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790722213368738-14927-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790722213368738-14927-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790722213368738-14927-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.147.253:40121
--webserver_interface=127.14.147.253
--webserver_port=0
--keytab_file=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790722213368738-14927-0/krb5kdc/oryx_127.14.147.253.keytab
--principal=oryx/127.14.147.253
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.147.212:34343
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.147.254:45601,127.14.147.253:40121
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790722213368738-14927-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790722213368738-14927-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790722213368738-14927-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 22:50:39.155140 21481 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 22:50:39.155369 21481 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 22:50:39.155393 21481 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 22:50:39.156639 21481 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 22:50:39.156695 21481 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 22:50:39.156709 21481 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 22:50:39.156723 21481 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 22:50:39.156738 21481 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 22:50:39.158275 21481 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.147.212:34343
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790722213368738-14927-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790722213368738-14927-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.14.147.254:45601,127.14.147.253:40121
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790722213368738-14927-0/krb5kdc/oryx_127.14.147.253.keytab
--principal=oryx/127.14.147.253
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.147.253:40121
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790722213368738-14927-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.14.147.253
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21481
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790722213368738-14927-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 21:58:54 UTC on ccycloud.quasar-xguoti.root.comops.site
build id 4681390
W20260929 22:50:39.158390 21481 master_options.cc:55] Only 2 masters are specified by master_addresses_flag ('127.14.147.254:45601,127.14.147.253:40121'), but minimum of 3 are required to tolerate failures of any one master. It is recommended to use at least 3 masters.
I20260929 22:50:39.158507 21481 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 22:50:39.158703 21481 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 22:50:39.163273 21487 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 22:50:39.163292 21489 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 22:50:39.163281 21486 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 22:50:39.163590 21481 server_base.cc:1034] running on GCE node
I20260929 22:50:39.163815 21481 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 22:50:39.164125 21481 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 22:50:39.165270 21481 hybrid_clock.cc:648] HybridClock initialized: now 1790722239165251 us; error 34 us; skew 500 ppm
Sep 29 22:50:39 dist-test-slave-98mk krb5kdc[20968](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790722239, etypes {rep=17 tkt=17 ses=17}, oryx/127.14.147.253@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 22:50:39.167719 21481 init.cc:377] Logged in from keytab as oryx/127.14.147.253@KRBTEST.COM (short username oryx)
I20260929 22:50:39.168169 21481 webserver.cc:460] Webserver started at http://127.14.147.253:34515/ using document root <none> and password file <none>
I20260929 22:50:39.168378 21481 fs_manager.cc:359] Metadata directory not provided
I20260929 22:50:39.168426 21481 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 22:50:39.168547 21481 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 22:50:39.169595 21481 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790722213368738-14927-0/minicluster-data/master-1/data/instance:
uuid: "e820b147283e4503b216af02b09dce4a"
format_stamp: "Formatted at 2026-09-29 22:50:39 on dist-test-slave-98mk"
server_key: "bd3ade942fa350b63d3c510ac5ebc8b3"
server_key_iv: "52c1f391b8f3d26e0c72eebf6e02d2d3"
server_key_version: "encryptionkey@0"
I20260929 22:50:39.169930 21481 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790722213368738-14927-0/minicluster-data/master-1/wal/instance:
uuid: "e820b147283e4503b216af02b09dce4a"
format_stamp: "Formatted at 2026-09-29 22:50:39 on dist-test-slave-98mk"
server_key: "bd3ade942fa350b63d3c510ac5ebc8b3"
server_key_iv: "52c1f391b8f3d26e0c72eebf6e02d2d3"
server_key_version: "encryptionkey@0"
I20260929 22:50:39.171269 21481 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260929 22:50:39.172043 21496 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 22:50:39.172222 21481 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260929 22:50:39.172304 21481 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790722213368738-14927-0/minicluster-data/master-1/data,/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790722213368738-14927-0/minicluster-data/master-1/wal
uuid: "e820b147283e4503b216af02b09dce4a"
format_stamp: "Formatted at 2026-09-29 22:50:39 on dist-test-slave-98mk"
server_key: "bd3ade942fa350b63d3c510ac5ebc8b3"
server_key_iv: "52c1f391b8f3d26e0c72eebf6e02d2d3"
server_key_version: "encryptionkey@0"
I20260929 22:50:39.172374 21481 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790722213368738-14927-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790722213368738-14927-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790722213368738-14927-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 22:50:39.184763 21481 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 22:50:39.186748 21481 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 22:50:39.186888 21481 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 22:50:39.190620 21481 rpc_server.cc:307] RPC server started. Bound to: 127.14.147.253:40121
I20260929 22:50:39.190677 21549 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.147.253:40121 every 8 connection(s)
I20260929 22:50:39.190990 21481 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790722213368738-14927-0/minicluster-data/master-1/data/info.pb
I20260929 22:50:39.191955 21550 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 22:50:39.193459 21550 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.14.147.254" port: 45601 } has no permanent_uuid. Determining permanent_uuid...
I20260929 22:50:39.198453 14927 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhWOCLk/build/release/bin/kudu as pid 21481
Sep 29 22:50:39 dist-test-slave-98mk krb5kdc[20968](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790722239, etypes {rep=17 tkt=17 ses=17}, oryx/127.14.147.253@KRBTEST.COM for oryx/127.14.147.254@KRBTEST.COM
I20260929 22:50:39.201905 21073 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:50:39.194116 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53118 (local address 127.14.147.254:45601)
0929 22:50:39.194167 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0929 22:50:39.194168 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0929 22:50:39.194346 (+ 178us) server_negotiation.cc:407] Connection header received
0929 22:50:39.194823 (+ 477us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 22:50:39.194825 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 22:50:39.194836 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 22:50:39.194899 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0929 22:50:39.195514 (+ 615us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:50:39.196002 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:50:39.197022 (+ 1020us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:50:39.197235 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:50:39.197293 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 22:50:39.199387 (+ 2094us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 22:50:39.199388 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 22:50:39.199389 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 22:50:39.199395 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0929 22:50:39.200636 (+ 1241us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 22:50:39.200998 (+ 362us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 22:50:39.200999 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 22:50:39.200999 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 22:50:39.201053 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 22:50:39.201271 (+ 218us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 22:50:39.201272 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 22:50:39.201272 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 22:50:39.201422 (+ 150us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 22:50:39.201479 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0929 22:50:39.201736 (+ 257us) server_negotiation.cc:299] Negotiation successful
0929 22:50:39.201817 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260929 22:50:39.202064 21552 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:50:39.194075 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.147.254:45601 (local address 127.0.0.1:53118)
0929 22:50:39.194234 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0929 22:50:39.194276 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0929 22:50:39.194717 (+ 441us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 22:50:39.194958 (+ 241us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 22:50:39.194961 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 22:50:39.195213 (+ 252us) client_negotiation.cc:189] Negotiated authn=SASL
0929 22:50:39.195402 (+ 189us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 22:50:39.195406 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 22:50:39.196075 (+ 669us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 22:50:39.196076 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 22:50:39.196943 (+ 867us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 22:50:39.196946 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 22:50:39.197318 (+ 372us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 22:50:39.197319 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 22:50:39.197431 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 22:50:39.197932 (+ 501us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 22:50:39.197940 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0929 22:50:39.199305 (+ 1365us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 22:50:39.200712 (+ 1407us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 22:50:39.200720 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 22:50:39.200722 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 22:50:39.200934 (+ 212us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 22:50:39.201104 (+ 170us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 22:50:39.201105 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 22:50:39.201105 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 22:50:39.201217 (+ 112us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 22:50:39.201478 (+ 261us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 22:50:39.201484 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 22:50:39.201617 (+ 133us) client_negotiation.cc:769] Sending connection context
0929 22:50:39.201708 (+ 91us) client_negotiation.cc:240] Negotiation successful
0929 22:50:39.201854 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":84,"threads_started":1}
I20260929 22:50:39.202713 21550 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.14.147.253" port: 40121 } has no permanent_uuid. Determining permanent_uuid...
Sep 29 22:50:39 dist-test-slave-98mk krb5kdc[20968](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790722238, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.14.147.253@KRBTEST.COM
Sep 29 22:50:39 dist-test-slave-98mk krb5kdc[20968](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790722239, etypes {rep=17 tkt=17 ses=17}, oryx/127.14.147.253@KRBTEST.COM for oryx/127.14.147.253@KRBTEST.COM
I20260929 22:50:39.206382 21554 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:50:39.198907 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56722 (local address 127.14.147.253:40121)
0929 22:50:39.199145 (+ 238us) server_negotiation.cc:206] Beginning negotiation
0929 22:50:39.199149 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 22:50:39.199166 (+ 17us) server_negotiation.cc:407] Connection header received
0929 22:50:39.199183 (+ 17us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 22:50:39.199184 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 22:50:39.199200 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 22:50:39.199269 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0929 22:50:39.199821 (+ 552us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:50:39.200497 (+ 676us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:50:39.201167 (+ 670us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:50:39.201483 (+ 316us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:50:39.201537 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 22:50:39.203788 (+ 2251us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 22:50:39.203797 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 22:50:39.203797 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 22:50:39.203804 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0929 22:50:39.205218 (+ 1414us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 22:50:39.205572 (+ 354us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 22:50:39.205575 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 22:50:39.205575 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 22:50:39.205631 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 22:50:39.205815 (+ 184us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 22:50:39.205816 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 22:50:39.205816 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 22:50:39.205987 (+ 171us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 22:50:39.206148 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0929 22:50:39.206190 (+ 42us) server_negotiation.cc:299] Negotiation successful
0929 22:50:39.206274 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":212,"thread_start_us":80,"threads_started":1}
I20260929 22:50:39.209204 21552 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:50:39.202945 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.147.253:40121 (local address 127.0.0.1:56724)
0929 22:50:39.202983 (+ 38us) negotiation.cc:107] Waiting for socket to connect
0929 22:50:39.202992 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0929 22:50:39.203024 (+ 32us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 22:50:39.203242 (+ 218us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 22:50:39.203242 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 22:50:39.203378 (+ 136us) client_negotiation.cc:189] Negotiated authn=SASL
0929 22:50:39.203459 (+ 81us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 22:50:39.203461 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 22:50:39.204108 (+ 647us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 22:50:39.204109 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 22:50:39.204571 (+ 462us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 22:50:39.204572 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 22:50:39.204900 (+ 328us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 22:50:39.204901 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 22:50:39.204959 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 22:50:39.205284 (+ 325us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 22:50:39.205288 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0929 22:50:39.206607 (+ 1319us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 22:50:39.208392 (+ 1785us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 22:50:39.208393 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 22:50:39.208393 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 22:50:39.208540 (+ 147us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 22:50:39.208677 (+ 137us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 22:50:39.208677 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 22:50:39.208678 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 22:50:39.208719 (+ 41us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 22:50:39.208932 (+ 213us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 22:50:39.208933 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 22:50:39.208974 (+ 41us) client_negotiation.cc:769] Sending connection context
0929 22:50:39.209039 (+ 65us) client_negotiation.cc:240] Negotiation successful
0929 22:50:39.209117 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":25}
I20260929 22:50:39.209287 21555 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:50:39.202998 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56724 (local address 127.14.147.253:40121)
0929 22:50:39.203146 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0929 22:50:39.203148 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 22:50:39.203152 (+ 4us) server_negotiation.cc:407] Connection header received
0929 22:50:39.203166 (+ 14us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 22:50:39.203166 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 22:50:39.203180 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 22:50:39.203233 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0929 22:50:39.203532 (+ 299us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:50:39.204041 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:50:39.204636 (+ 595us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:50:39.204847 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:50:39.204898 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 22:50:39.206705 (+ 1807us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 22:50:39.206706 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 22:50:39.206706 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 22:50:39.206712 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0929 22:50:39.208326 (+ 1614us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 22:50:39.208595 (+ 269us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 22:50:39.208596 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 22:50:39.208596 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 22:50:39.208638 (+ 42us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 22:50:39.208751 (+ 113us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 22:50:39.208752 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 22:50:39.208752 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 22:50:39.208839 (+ 87us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 22:50:39.208881 (+ 42us) server_negotiation.cc:1035] Waiting for connection context
0929 22:50:39.209057 (+ 176us) server_negotiation.cc:299] Negotiation successful
0929 22:50:39.209158 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":46,"server-negotiator.queue_time_us":133,"thread_start_us":99,"threads_started":1}
I20260929 22:50:39.210462 21550 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e820b147283e4503b216af02b09dce4a: Bootstrap starting.
I20260929 22:50:39.211022 21550 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e820b147283e4503b216af02b09dce4a: Neither blocks nor log segments found. Creating new log.
I20260929 22:50:39.211247 21550 log.cc:824] T 00000000000000000000000000000000 P e820b147283e4503b216af02b09dce4a: Log is configured to *not* fsync() on all Append() calls
I20260929 22:50:39.211913 21550 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e820b147283e4503b216af02b09dce4a: No bootstrap required, opened a new log
I20260929 22:50:39.213307 21550 raft_consensus.cc:348] T 00000000000000000000000000000000 P e820b147283e4503b216af02b09dce4a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d38d0f6f0d3c4d2b8b7131e3f0f72ffe" member_type: VOTER last_known_addr { host: "127.14.147.254" port: 45601 } } peers { permanent_uuid: "e820b147283e4503b216af02b09dce4a" member_type: VOTER last_known_addr { host: "127.14.147.253" port: 40121 } }
I20260929 22:50:39.213433 21550 raft_consensus.cc:374] T 00000000000000000000000000000000 P e820b147283e4503b216af02b09dce4a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 22:50:39.213454 21550 raft_consensus.cc:729] T 00000000000000000000000000000000 P e820b147283e4503b216af02b09dce4a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e820b147283e4503b216af02b09dce4a, State: Initialized, Role: FOLLOWER
I20260929 22:50:39.213552 21550 consensus_queue.cc:260] T 00000000000000000000000000000000 P e820b147283e4503b216af02b09dce4a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d38d0f6f0d3c4d2b8b7131e3f0f72ffe" member_type: VOTER last_known_addr { host: "127.14.147.254" port: 45601 } } peers { permanent_uuid: "e820b147283e4503b216af02b09dce4a" member_type: VOTER last_known_addr { host: "127.14.147.253" port: 40121 } }
I20260929 22:50:39.213829 21550 sys_catalog.cc:564] T 00000000000000000000000000000000 P e820b147283e4503b216af02b09dce4a [sys.catalog]: configured and running, proceeding with master startup.
I20260929 22:50:39.213804 21557 sys_catalog.cc:455] T 00000000000000000000000000000000 P e820b147283e4503b216af02b09dce4a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d38d0f6f0d3c4d2b8b7131e3f0f72ffe" member_type: VOTER last_known_addr { host: "127.14.147.254" port: 45601 } } peers { permanent_uuid: "e820b147283e4503b216af02b09dce4a" member_type: VOTER last_known_addr { host: "127.14.147.253" port: 40121 } } }
I20260929 22:50:39.213881 21557 sys_catalog.cc:458] T 00000000000000000000000000000000 P e820b147283e4503b216af02b09dce4a [sys.catalog]: This master's current role is: FOLLOWER
W20260929 22:50:39.215390 21568 catalog_manager.cc:1558] T 00000000000000000000000000000000 P e820b147283e4503b216af02b09dce4a: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 22:50:39.215471 21568 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260929 22:50:39.217059 21481 master_runner.cc:428] Detected that this master e820b147283e4503b216af02b09dce4a is joining an existing cluster
I20260929 22:50:39.217100 21481 master_runner.cc:432] Initiating AddMaster RPC to add 127.14.147.253:40121
I20260929 22:50:39.224845 21555 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:50:39.218177 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56736 (local address 127.14.147.253:40121)
0929 22:50:39.218213 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0929 22:50:39.218215 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 22:50:39.218375 (+ 160us) server_negotiation.cc:407] Connection header received
0929 22:50:39.218399 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 22:50:39.218399 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 22:50:39.218408 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 22:50:39.218452 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0929 22:50:39.218951 (+ 499us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:50:39.219760 (+ 809us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:50:39.220833 (+ 1073us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:50:39.221045 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:50:39.221100 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 22:50:39.222209 (+ 1109us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 22:50:39.222210 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 22:50:39.222211 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 22:50:39.222217 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0929 22:50:39.223738 (+ 1521us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 22:50:39.224103 (+ 365us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 22:50:39.224104 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 22:50:39.224104 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 22:50:39.224158 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 22:50:39.224327 (+ 169us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 22:50:39.224328 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 22:50:39.224328 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 22:50:39.224450 (+ 122us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 22:50:39.224514 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0929 22:50:39.224647 (+ 133us) server_negotiation.cc:299] Negotiation successful
0929 22:50:39.224731 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260929 22:50:39.225034 21073 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:50:39.218081 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53124 (local address 127.14.147.254:45601)
0929 22:50:39.218129 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0929 22:50:39.218131 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 22:50:39.218247 (+ 116us) server_negotiation.cc:407] Connection header received
0929 22:50:39.218300 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 22:50:39.218301 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 22:50:39.218313 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 22:50:39.218376 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0929 22:50:39.219498 (+ 1122us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:50:39.220148 (+ 650us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:50:39.221186 (+ 1038us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:50:39.221439 (+ 253us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:50:39.221503 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 22:50:39.222485 (+ 982us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 22:50:39.222487 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 22:50:39.222487 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 22:50:39.222495 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0929 22:50:39.223815 (+ 1320us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 22:50:39.224171 (+ 356us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 22:50:39.224172 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 22:50:39.224173 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 22:50:39.224229 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 22:50:39.224406 (+ 177us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 22:50:39.224407 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 22:50:39.224408 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 22:50:39.224548 (+ 140us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 22:50:39.224619 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0929 22:50:39.224738 (+ 119us) server_negotiation.cc:299] Negotiation successful
0929 22:50:39.224818 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260929 22:50:39.224912 21575 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:50:39.217994 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.147.254:45601 (local address 127.0.0.1:53124)
0929 22:50:39.218185 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0929 22:50:39.218196 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0929 22:50:39.218251 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 22:50:39.218487 (+ 236us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 22:50:39.218488 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 22:50:39.218757 (+ 269us) client_negotiation.cc:189] Negotiated authn=SASL
0929 22:50:39.218870 (+ 113us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 22:50:39.218872 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 22:50:39.220233 (+ 1361us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 22:50:39.220234 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 22:50:39.221071 (+ 837us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 22:50:39.221073 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 22:50:39.221510 (+ 437us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 22:50:39.221511 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 22:50:39.221578 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 22:50:39.221917 (+ 339us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 22:50:39.221922 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0929 22:50:39.222359 (+ 437us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 22:50:39.223889 (+ 1530us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 22:50:39.223890 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 22:50:39.223890 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 22:50:39.224111 (+ 221us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 22:50:39.224283 (+ 172us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 22:50:39.224283 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 22:50:39.224284 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 22:50:39.224352 (+ 68us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 22:50:39.224617 (+ 265us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 22:50:39.224617 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 22:50:39.224648 (+ 31us) client_negotiation.cc:769] Sending connection context
0929 22:50:39.224701 (+ 53us) client_negotiation.cc:240] Negotiation successful
0929 22:50:39.224785 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":169,"mutex_wait_us":2,"thread_start_us":93,"threads_started":1}
I20260929 22:50:39.225366 21574 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:50:39.218013 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.147.253:40121 (local address 127.0.0.1:56736)
0929 22:50:39.218318 (+ 305us) negotiation.cc:107] Waiting for socket to connect
0929 22:50:39.218325 (+ 7us) client_negotiation.cc:174] Beginning negotiation
0929 22:50:39.218366 (+ 41us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 22:50:39.218454 (+ 88us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 22:50:39.218454 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 22:50:39.218760 (+ 306us) client_negotiation.cc:189] Negotiated authn=SASL
0929 22:50:39.218870 (+ 110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 22:50:39.218872 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 22:50:39.219837 (+ 965us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 22:50:39.219838 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 22:50:39.220749 (+ 911us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 22:50:39.220751 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 22:50:39.221118 (+ 367us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 22:50:39.221119 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 22:50:39.221176 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 22:50:39.221701 (+ 525us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 22:50:39.221705 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0929 22:50:39.222138 (+ 433us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 22:50:39.223821 (+ 1683us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 22:50:39.223822 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 22:50:39.223823 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 22:50:39.224044 (+ 221us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 22:50:39.224212 (+ 168us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 22:50:39.224213 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 22:50:39.224213 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 22:50:39.224274 (+ 61us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 22:50:39.224509 (+ 235us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 22:50:39.224510 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 22:50:39.224549 (+ 39us) client_negotiation.cc:769] Sending connection context
0929 22:50:39.224943 (+ 394us) client_negotiation.cc:240] Negotiation successful
0929 22:50:39.225020 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":295,"mutex_wait_us":16,"thread_start_us":85,"threads_started":1}
Sep 29 22:50:39 dist-test-slave-98mk krb5kdc[20968](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790722238, etypes {rep=17 tkt=17 ses=17}, oryx/127.14.147.254@KRBTEST.COM for oryx/127.14.147.253@KRBTEST.COM
I20260929 22:50:39.234755 21576 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:50:39.227349 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.147.253:40121 (local address 127.0.0.1:56752)
0929 22:50:39.227535 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0929 22:50:39.227588 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0929 22:50:39.227662 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 22:50:39.227804 (+ 142us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 22:50:39.227807 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 22:50:39.228143 (+ 336us) client_negotiation.cc:189] Negotiated authn=SASL
0929 22:50:39.228208 (+ 65us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 22:50:39.228210 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 22:50:39.228807 (+ 597us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 22:50:39.228808 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 22:50:39.229749 (+ 941us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 22:50:39.229752 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 22:50:39.230075 (+ 323us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 22:50:39.230076 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 22:50:39.230167 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 22:50:39.230586 (+ 419us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 22:50:39.230591 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0929 22:50:39.232086 (+ 1495us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 22:50:39.233681 (+ 1595us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 22:50:39.233691 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 22:50:39.233693 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 22:50:39.233913 (+ 220us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 22:50:39.234110 (+ 197us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 22:50:39.234111 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 22:50:39.234111 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 22:50:39.234172 (+ 61us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 22:50:39.234443 (+ 271us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 22:50:39.234450 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 22:50:39.234503 (+ 53us) client_negotiation.cc:769] Sending connection context
0929 22:50:39.234573 (+ 70us) client_negotiation.cc:240] Negotiation successful
0929 22:50:39.234679 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":151,"thread_start_us":100,"threads_started":1}
I20260929 22:50:39.234809 21555 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:50:39.227445 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56752 (local address 127.14.147.253:40121)
0929 22:50:39.227536 (+ 91us) server_negotiation.cc:206] Beginning negotiation
0929 22:50:39.227537 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0929 22:50:39.227638 (+ 101us) server_negotiation.cc:407] Connection header received
0929 22:50:39.227742 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 22:50:39.227743 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 22:50:39.227750 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 22:50:39.227797 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0929 22:50:39.228286 (+ 489us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:50:39.228740 (+ 454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:50:39.229830 (+ 1090us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:50:39.230011 (+ 181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:50:39.230064 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 22:50:39.232169 (+ 2105us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 22:50:39.232171 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 22:50:39.232171 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 22:50:39.232179 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0929 22:50:39.233554 (+ 1375us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 22:50:39.233982 (+ 428us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 22:50:39.233983 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 22:50:39.233983 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 22:50:39.234042 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 22:50:39.234234 (+ 192us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 22:50:39.234235 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 22:50:39.234235 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 22:50:39.234371 (+ 136us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 22:50:39.234437 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0929 22:50:39.234621 (+ 184us) server_negotiation.cc:299] Negotiation successful
0929 22:50:39.234714 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260929 22:50:39.235657 21014 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.14.147.253:40121: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "e820b147283e4503b216af02b09dce4a"
member_type: NON_VOTER
last_known_addr {
host: "127.14.147.253"
port: 40121
}
attrs {
promote: true
}
}
dest_uuid: "d38d0f6f0d3c4d2b8b7131e3f0f72ffe"
cas_config_opid_index: -1
I20260929 22:50:39.235958 21014 consensus_queue.cc:237] T 00000000000000000000000000000000 P d38d0f6f0d3c4d2b8b7131e3f0f72ffe [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "d38d0f6f0d3c4d2b8b7131e3f0f72ffe" member_type: VOTER last_known_addr { host: "127.14.147.254" port: 45601 } } peers { permanent_uuid: "e820b147283e4503b216af02b09dce4a" member_type: NON_VOTER last_known_addr { host: "127.14.147.253" port: 40121 } attrs { promote: true } }
I20260929 22:50:39.236442 21578 raft_consensus.cc:2932] T 00000000000000000000000000000000 P d38d0f6f0d3c4d2b8b7131e3f0f72ffe [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER e820b147283e4503b216af02b09dce4a (127.14.147.253) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "d38d0f6f0d3c4d2b8b7131e3f0f72ffe" member_type: VOTER last_known_addr { host: "127.14.147.254" port: 45601 } } peers { permanent_uuid: "e820b147283e4503b216af02b09dce4a" member_type: NON_VOTER last_known_addr { host: "127.14.147.253" port: 40121 } attrs { promote: true } } }
I20260929 22:50:39.236752 21524 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e820b147283e4503b216af02b09dce4a [term 0 FOLLOWER]: Advancing to term 1
I20260929 22:50:39.237525 21578 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.14.147.253:40121
I20260929 22:50:39.237612 21524 raft_consensus.cc:1261] T 00000000000000000000000000000000 P e820b147283e4503b216af02b09dce4a [term 1 FOLLOWER]: Refusing update from remote peer d38d0f6f0d3c4d2b8b7131e3f0f72ffe: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20260929 22:50:39.237614 21578 sys_catalog.cc:455] T 00000000000000000000000000000000 P d38d0f6f0d3c4d2b8b7131e3f0f72ffe [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "d38d0f6f0d3c4d2b8b7131e3f0f72ffe" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "d38d0f6f0d3c4d2b8b7131e3f0f72ffe" member_type: VOTER last_known_addr { host: "127.14.147.254" port: 45601 } } peers { permanent_uuid: "e820b147283e4503b216af02b09dce4a" member_type: NON_VOTER last_known_addr { host: "127.14.147.253" port: 40121 } attrs { promote: true } } }
I20260929 22:50:39.237668 21578 sys_catalog.cc:458] T 00000000000000000000000000000000 P d38d0f6f0d3c4d2b8b7131e3f0f72ffe [sys.catalog]: This master's current role is: LEADER
I20260929 22:50:39.237779 21481 master.cc:537] Master@127.14.147.253:40121 shutting down...
I20260929 22:50:39.238001 21577 consensus_queue.cc:1035] T 00000000000000000000000000000000 P d38d0f6f0d3c4d2b8b7131e3f0f72ffe [LEADER]: Connected to new peer: Peer: permanent_uuid: "e820b147283e4503b216af02b09dce4a" member_type: NON_VOTER last_known_addr { host: "127.14.147.253" port: 40121 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 0, Time since last communication: 0.000s
W20260929 22:50:39.238852 20998 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master
W20260929 22:50:39.239331 20998 consensus_peers.cc:487] T 00000000000000000000000000000000 P d38d0f6f0d3c4d2b8b7131e3f0f72ffe -> Peer e820b147283e4503b216af02b09dce4a (127.14.147.253:40121): Couldn't send request to peer e820b147283e4503b216af02b09dce4a. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260929 22:50:39.240429 21481 raft_consensus.cc:2229] T 00000000000000000000000000000000 P e820b147283e4503b216af02b09dce4a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 22:50:39.240520 21481 raft_consensus.cc:2258] T 00000000000000000000000000000000 P e820b147283e4503b216af02b09dce4a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 22:50:39.240573 21481 tablet_replica.cc:330] T 00000000000000000000000000000000 P e820b147283e4503b216af02b09dce4a: stopping tablet replica
I20260929 22:50:39.253602 21481 master.cc:552] Master@127.14.147.253:40121 shutdown complete.
I20260929 22:50:39.253676 21481 master_runner.cc:305] Clearing existing system tablet
I20260929 22:50:39.253811 21481 fs_manager.cc:359] Metadata directory not provided
I20260929 22:50:39.253859 21481 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 22:50:39.254558 21481 fs_manager.cc:709] Time spent opening directory manager: real 0.000s user 0.002s sys 0.000s
I20260929 22:50:39.255079 21582 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 22:50:39.255203 21481 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790722213368738-14927-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 22:50:39.255264 21481 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260929 22:50:39.255287 21481 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790722213368738-14927-0/minicluster-data/master-1/data,/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790722213368738-14927-0/minicluster-data/master-1/wal
uuid: "e820b147283e4503b216af02b09dce4a"
format_stamp: "Formatted at 2026-09-29 22:50:39 on dist-test-slave-98mk"
server_key: "bd3ade942fa350b63d3c510ac5ebc8b3"
server_key_iv: "52c1f391b8f3d26e0c72eebf6e02d2d3"
server_key_version: "encryptionkey@0"
I20260929 22:50:39.256163 21481 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P e820b147283e4503b216af02b09dce4a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 22:50:39.257807 21481 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P e820b147283e4503b216af02b09dce4a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260929 22:50:39.257902 21481 log.cc:1196] T 00000000000000000000000000000000 P e820b147283e4503b216af02b09dce4a: Deleting WAL directory at /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790722213368738-14927-0/minicluster-data/master-1/wal/wals/00000000000000000000000000000000
I20260929 22:50:39.258102 21481 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P e820b147283e4503b216af02b09dce4a: Deleting consensus metadata
I20260929 22:50:39.258212 21481 master_runner.cc:315] Copying system tablet from 127.14.147.254:45601
I20260929 22:50:39.258744 21481 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P e820b147283e4503b216af02b09dce4a: tablet copy: Beginning tablet copy session from remote peer at address 127.14.147.254:45601
I20260929 22:50:39.264285 21073 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:50:39.259104 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53126 (local address 127.14.147.254:45601)
0929 22:50:39.259155 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0929 22:50:39.259157 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 22:50:39.259240 (+ 83us) server_negotiation.cc:407] Connection header received
0929 22:50:39.259302 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 22:50:39.259303 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 22:50:39.259312 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 22:50:39.259361 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0929 22:50:39.259685 (+ 324us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:50:39.260174 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:50:39.260903 (+ 729us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:50:39.261088 (+ 185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:50:39.261143 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 22:50:39.262015 (+ 872us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 22:50:39.262016 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 22:50:39.262017 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 22:50:39.262024 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0929 22:50:39.263216 (+ 1192us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 22:50:39.263577 (+ 361us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 22:50:39.263578 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 22:50:39.263578 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 22:50:39.263632 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 22:50:39.263801 (+ 169us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 22:50:39.263802 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 22:50:39.263803 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 22:50:39.263914 (+ 111us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 22:50:39.263967 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0929 22:50:39.264100 (+ 133us) server_negotiation.cc:299] Negotiation successful
0929 22:50:39.264179 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260929 22:50:39.264299 21588 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:50:39.259033 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.147.254:45601 (local address 127.0.0.1:53126)
0929 22:50:39.259156 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0929 22:50:39.259166 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0929 22:50:39.259240 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 22:50:39.259365 (+ 125us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 22:50:39.259365 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 22:50:39.259552 (+ 187us) client_negotiation.cc:189] Negotiated authn=SASL
0929 22:50:39.259614 (+ 62us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 22:50:39.259616 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 22:50:39.260251 (+ 635us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 22:50:39.260252 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 22:50:39.260827 (+ 575us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 22:50:39.260830 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 22:50:39.261149 (+ 319us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 22:50:39.261150 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 22:50:39.261210 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 22:50:39.261625 (+ 415us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 22:50:39.261629 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0929 22:50:39.261927 (+ 298us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 22:50:39.263303 (+ 1376us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 22:50:39.263304 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 22:50:39.263304 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 22:50:39.263521 (+ 217us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 22:50:39.263684 (+ 163us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 22:50:39.263685 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 22:50:39.263685 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 22:50:39.263752 (+ 67us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 22:50:39.263964 (+ 212us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 22:50:39.263965 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 22:50:39.264013 (+ 48us) client_negotiation.cc:769] Sending connection context
0929 22:50:39.264071 (+ 58us) client_negotiation.cc:240] Negotiation successful
0929 22:50:39.264181 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":99,"thread_start_us":78,"threads_started":1}
I20260929 22:50:39.264808 21034 tablet_copy_service.cc:140] P d38d0f6f0d3c4d2b8b7131e3f0f72ffe: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer e820b147283e4503b216af02b09dce4a ({username='oryx', principal='oryx/127.14.147.253@KRBTEST.COM'} at 127.0.0.1:53126)
I20260929 22:50:39.264894 21034 tablet_copy_service.cc:161] P d38d0f6f0d3c4d2b8b7131e3f0f72ffe: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer e820b147283e4503b216af02b09dce4a at {username='oryx', principal='oryx/127.14.147.253@KRBTEST.COM'} at 127.0.0.1:53126: session id = e820b147283e4503b216af02b09dce4a-00000000000000000000000000000000
I20260929 22:50:39.265527 21034 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P d38d0f6f0d3c4d2b8b7131e3f0f72ffe: Tablet Copy: opened 0 blocks and 1 log segments
I20260929 22:50:39.266175 21481 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260929 22:50:39.267211 21481 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P e820b147283e4503b216af02b09dce4a: tablet copy: Starting download of 0 data blocks...
I20260929 22:50:39.267356 21481 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P e820b147283e4503b216af02b09dce4a: tablet copy: Starting download of 1 WAL segments...
I20260929 22:50:39.268210 21481 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P e820b147283e4503b216af02b09dce4a: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260929 22:50:39.268985 21034 tablet_copy_service.cc:342] P d38d0f6f0d3c4d2b8b7131e3f0f72ffe: Request end of tablet copy session e820b147283e4503b216af02b09dce4a-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.14.147.253@KRBTEST.COM'} at 127.0.0.1:53126
I20260929 22:50:39.269037 21034 tablet_copy_service.cc:433] P d38d0f6f0d3c4d2b8b7131e3f0f72ffe: ending tablet copy session e820b147283e4503b216af02b09dce4a-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer e820b147283e4503b216af02b09dce4a
I20260929 22:50:39.271530 21481 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 22:50:39.272643 21589 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 22:50:39.272722 21590 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 22:50:39.272828 21481 server_base.cc:1034] running on GCE node
W20260929 22:50:39.273216 21592 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 22:50:39.273345 21481 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 22:50:39.273514 21481 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 22:50:39.274648 21481 hybrid_clock.cc:648] HybridClock initialized: now 1790722239274641 us; error 26 us; skew 500 ppm
Sep 29 22:50:39 dist-test-slave-98mk krb5kdc[20968](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790722239, etypes {rep=17 tkt=17 ses=17}, oryx/127.14.147.253@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 22:50:39.276103 21481 init.cc:377] Logged in from keytab as oryx/127.14.147.253@KRBTEST.COM (short username oryx)
I20260929 22:50:39.276356 21481 webserver.cc:460] Webserver started at http://127.14.147.253:35265/ using document root <none> and password file <none>
I20260929 22:50:39.276466 21481 fs_manager.cc:359] Metadata directory not provided
I20260929 22:50:39.276497 21481 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 22:50:39.277215 21481 fs_manager.cc:709] Time spent opening directory manager: real 0.000s user 0.001s sys 0.000s
I20260929 22:50:39.277793 21599 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 22:50:39.277913 21481 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260929 22:50:39.277951 21481 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790722213368738-14927-0/minicluster-data/master-1/data,/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790722213368738-14927-0/minicluster-data/master-1/wal
uuid: "e820b147283e4503b216af02b09dce4a"
format_stamp: "Formatted at 2026-09-29 22:50:39 on dist-test-slave-98mk"
server_key: "bd3ade942fa350b63d3c510ac5ebc8b3"
server_key_iv: "52c1f391b8f3d26e0c72eebf6e02d2d3"
server_key_version: "encryptionkey@0"
I20260929 22:50:39.278009 21481 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790722213368738-14927-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790722213368738-14927-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790722213368738-14927-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 22:50:39.295662 21481 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 22:50:39.297137 21481 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 22:50:39.299544 21481 rpc_server.cc:307] RPC server started. Bound to: 127.14.147.253:40121
I20260929 22:50:39.299579 21652 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.147.253:40121 every 8 connection(s)
I20260929 22:50:39.299846 21481 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790722213368738-14927-0/minicluster-data/master-1/data/info.pb
I20260929 22:50:39.300725 21653 sys_catalog.cc:263] Verifying existing consensus state
I20260929 22:50:39.301299 21653 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e820b147283e4503b216af02b09dce4a: Bootstrap starting.
I20260929 22:50:39.304286 21653 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e820b147283e4503b216af02b09dce4a: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260929 22:50:39.304543 21653 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e820b147283e4503b216af02b09dce4a: Bootstrap complete.
I20260929 22:50:39.304744 21653 raft_consensus.cc:348] T 00000000000000000000000000000000 P e820b147283e4503b216af02b09dce4a [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "d38d0f6f0d3c4d2b8b7131e3f0f72ffe" member_type: VOTER last_known_addr { host: "127.14.147.254" port: 45601 } } peers { permanent_uuid: "e820b147283e4503b216af02b09dce4a" member_type: NON_VOTER last_known_addr { host: "127.14.147.253" port: 40121 } attrs { promote: true } }
I20260929 22:50:39.304836 21653 raft_consensus.cc:729] T 00000000000000000000000000000000 P e820b147283e4503b216af02b09dce4a [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: e820b147283e4503b216af02b09dce4a, State: Initialized, Role: LEARNER
I20260929 22:50:39.304875 21653 consensus_queue.cc:260] T 00000000000000000000000000000000 P e820b147283e4503b216af02b09dce4a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "d38d0f6f0d3c4d2b8b7131e3f0f72ffe" member_type: VOTER last_known_addr { host: "127.14.147.254" port: 45601 } } peers { permanent_uuid: "e820b147283e4503b216af02b09dce4a" member_type: NON_VOTER last_known_addr { host: "127.14.147.253" port: 40121 } attrs { promote: true } }
I20260929 22:50:39.305148 21653 sys_catalog.cc:564] T 00000000000000000000000000000000 P e820b147283e4503b216af02b09dce4a [sys.catalog]: configured and running, proceeding with master startup.
I20260929 22:50:39.305150 21656 sys_catalog.cc:455] T 00000000000000000000000000000000 P e820b147283e4503b216af02b09dce4a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "d38d0f6f0d3c4d2b8b7131e3f0f72ffe" member_type: VOTER last_known_addr { host: "127.14.147.254" port: 45601 } } peers { permanent_uuid: "e820b147283e4503b216af02b09dce4a" member_type: NON_VOTER last_known_addr { host: "127.14.147.253" port: 40121 } attrs { promote: true } } }
I20260929 22:50:39.305289 21656 sys_catalog.cc:458] T 00000000000000000000000000000000 P e820b147283e4503b216af02b09dce4a [sys.catalog]: This master's current role is: LEARNER
I20260929 22:50:39.306694 21667 catalog_manager.cc:1259] Loaded cluster ID: eeb78789c1ae4d64a674c24edf4ad136
I20260929 22:50:39.306751 21667 catalog_manager.cc:1552] T 00000000000000000000000000000000 P e820b147283e4503b216af02b09dce4a: loading cluster ID for follower catalog manager: success
I20260929 22:50:39.307717 21667 catalog_manager.cc:1574] T 00000000000000000000000000000000 P e820b147283e4503b216af02b09dce4a: acquiring CA information for follower catalog manager: success
I20260929 22:50:39.307988 21667 catalog_manager.cc:1602] T 00000000000000000000000000000000 P e820b147283e4503b216af02b09dce4a: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260929 22:50:39.314036 21668 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:50:39.308264 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56776 (local address 127.14.147.253:40121)
0929 22:50:39.308356 (+ 92us) server_negotiation.cc:206] Beginning negotiation
0929 22:50:39.308358 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 22:50:39.308366 (+ 8us) server_negotiation.cc:407] Connection header received
0929 22:50:39.308389 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 22:50:39.308389 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 22:50:39.308400 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 22:50:39.308454 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0929 22:50:39.308968 (+ 514us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:50:39.309610 (+ 642us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:50:39.310301 (+ 691us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:50:39.310511 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:50:39.310564 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 22:50:39.311723 (+ 1159us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 22:50:39.311725 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 22:50:39.311725 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 22:50:39.311732 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0929 22:50:39.313023 (+ 1291us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 22:50:39.313329 (+ 306us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 22:50:39.313329 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 22:50:39.313330 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 22:50:39.313397 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 22:50:39.313557 (+ 160us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 22:50:39.313558 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 22:50:39.313558 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 22:50:39.313661 (+ 103us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 22:50:39.313747 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0929 22:50:39.313855 (+ 108us) server_negotiation.cc:299] Negotiation successful
0929 22:50:39.313939 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75,"thread_start_us":50,"threads_started":1}
WARNING: no policy specified for oryx/127.14.147.252@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.14.147.252@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.14.147.252 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790722213368738-14927-0/krb5kdc/oryx_127.14.147.252.keytab.
Entry for principal oryx/127.14.147.252 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790722213368738-14927-0/krb5kdc/oryx_127.14.147.252.keytab.
I20260929 22:50:39.341977 14927 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhWOCLk/build/release/bin/kudu
/tmp/dist-test-taskhWOCLk/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790722213368738-14927-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790722213368738-14927-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790722213368738-14927-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790722213368738-14927-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.147.252:41485
--webserver_interface=127.14.147.252
--webserver_port=0
--keytab_file=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790722213368738-14927-0/krb5kdc/oryx_127.14.147.252.keytab
--principal=oryx/127.14.147.252
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.147.212:34343
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.147.254:45601,127.14.147.253:40121,127.14.147.252:41485
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790722213368738-14927-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790722213368738-14927-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790722213368738-14927-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 22:50:39.432209 21671 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 22:50:39.432447 21671 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 22:50:39.432471 21671 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 22:50:39.433702 21671 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 22:50:39.433753 21671 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 22:50:39.433765 21671 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 22:50:39.433777 21671 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 22:50:39.433800 21671 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 22:50:39.435456 21671 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.147.212:34343
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790722213368738-14927-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790722213368738-14927-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.14.147.254:45601,127.14.147.253:40121,127.14.147.252:41485
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790722213368738-14927-0/krb5kdc/oryx_127.14.147.252.keytab
--principal=oryx/127.14.147.252
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.147.252:41485
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790722213368738-14927-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.14.147.252
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21671
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790722213368738-14927-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 21:58:54 UTC on ccycloud.quasar-xguoti.root.comops.site
build id 4681390
I20260929 22:50:39.435725 21671 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 22:50:39.435961 21671 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 22:50:39.440820 21676 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 22:50:39.440946 21671 server_base.cc:1034] running on GCE node
W20260929 22:50:39.440801 21679 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 22:50:39.440801 21677 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 22:50:39.441303 21671 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 22:50:39.441617 21671 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 22:50:39.442772 21671 hybrid_clock.cc:648] HybridClock initialized: now 1790722239442756 us; error 32 us; skew 500 ppm
Sep 29 22:50:39 dist-test-slave-98mk krb5kdc[20968](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790722239, etypes {rep=17 tkt=17 ses=17}, oryx/127.14.147.252@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 22:50:39.445218 21671 init.cc:377] Logged in from keytab as oryx/127.14.147.252@KRBTEST.COM (short username oryx)
I20260929 22:50:39.445660 21671 webserver.cc:460] Webserver started at http://127.14.147.252:42309/ using document root <none> and password file <none>
I20260929 22:50:39.445875 21671 fs_manager.cc:359] Metadata directory not provided
I20260929 22:50:39.445923 21671 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 22:50:39.446046 21671 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 22:50:39.447151 21671 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790722213368738-14927-0/minicluster-data/master-2/data/instance:
uuid: "98a8b44ee91645f8a3dc6f02d1c5c698"
format_stamp: "Formatted at 2026-09-29 22:50:39 on dist-test-slave-98mk"
server_key: "887e4c07ce432cbb9d939f466c01d244"
server_key_iv: "82d69a50576c0fe675c6948659dd547c"
server_key_version: "encryptionkey@0"
I20260929 22:50:39.447541 21671 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790722213368738-14927-0/minicluster-data/master-2/wal/instance:
uuid: "98a8b44ee91645f8a3dc6f02d1c5c698"
format_stamp: "Formatted at 2026-09-29 22:50:39 on dist-test-slave-98mk"
server_key: "887e4c07ce432cbb9d939f466c01d244"
server_key_iv: "82d69a50576c0fe675c6948659dd547c"
server_key_version: "encryptionkey@0"
I20260929 22:50:39.448837 21671 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260929 22:50:39.449515 21686 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 22:50:39.449734 21671 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260929 22:50:39.449821 21671 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790722213368738-14927-0/minicluster-data/master-2/data,/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790722213368738-14927-0/minicluster-data/master-2/wal
uuid: "98a8b44ee91645f8a3dc6f02d1c5c698"
format_stamp: "Formatted at 2026-09-29 22:50:39 on dist-test-slave-98mk"
server_key: "887e4c07ce432cbb9d939f466c01d244"
server_key_iv: "82d69a50576c0fe675c6948659dd547c"
server_key_version: "encryptionkey@0"
I20260929 22:50:39.449891 21671 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790722213368738-14927-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790722213368738-14927-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790722213368738-14927-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 22:50:39.476250 21671 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 22:50:39.478454 21671 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 22:50:39.478601 21671 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 22:50:39.482687 21671 rpc_server.cc:307] RPC server started. Bound to: 127.14.147.252:41485
I20260929 22:50:39.482748 21739 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.147.252:41485 every 8 connection(s)
I20260929 22:50:39.483069 21671 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790722213368738-14927-0/minicluster-data/master-2/data/info.pb
I20260929 22:50:39.484074 21740 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 22:50:39.485730 21740 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.14.147.254" port: 45601 } has no permanent_uuid. Determining permanent_uuid...
I20260929 22:50:39.487610 14927 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhWOCLk/build/release/bin/kudu as pid 21671
Sep 29 22:50:39 dist-test-slave-98mk krb5kdc[20968](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790722239, etypes {rep=17 tkt=17 ses=17}, oryx/127.14.147.252@KRBTEST.COM for oryx/127.14.147.254@KRBTEST.COM
Sep 29 22:50:39 dist-test-slave-98mk krb5kdc[20968](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790722238, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.14.147.252@KRBTEST.COM
I20260929 22:50:39.495568 21073 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:50:39.486681 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53128 (local address 127.14.147.254:45601)
0929 22:50:39.486777 (+ 96us) server_negotiation.cc:206] Beginning negotiation
0929 22:50:39.486779 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 22:50:39.486793 (+ 14us) server_negotiation.cc:407] Connection header received
0929 22:50:39.487246 (+ 453us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 22:50:39.487248 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 22:50:39.487262 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 22:50:39.487308 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0929 22:50:39.488219 (+ 911us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:50:39.488757 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:50:39.489857 (+ 1100us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:50:39.490084 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:50:39.490134 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 22:50:39.492568 (+ 2434us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 22:50:39.492569 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 22:50:39.492570 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 22:50:39.492578 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0929 22:50:39.494047 (+ 1469us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 22:50:39.494554 (+ 507us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 22:50:39.494556 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 22:50:39.494556 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 22:50:39.494620 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 22:50:39.494877 (+ 257us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 22:50:39.494878 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 22:50:39.494878 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 22:50:39.495025 (+ 147us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 22:50:39.495095 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0929 22:50:39.495349 (+ 254us) server_negotiation.cc:299] Negotiation successful
0929 22:50:39.495456 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260929 22:50:39.495669 21742 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:50:39.486418 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.147.254:45601 (local address 127.0.0.1:53128)
0929 22:50:39.486598 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0929 22:50:39.486643 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0929 22:50:39.487138 (+ 495us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 22:50:39.487387 (+ 249us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 22:50:39.487390 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 22:50:39.487811 (+ 421us) client_negotiation.cc:189] Negotiated authn=SASL
0929 22:50:39.488075 (+ 264us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 22:50:39.488082 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 22:50:39.488889 (+ 807us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 22:50:39.488890 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 22:50:39.489758 (+ 868us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 22:50:39.489760 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 22:50:39.490201 (+ 441us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 22:50:39.490202 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 22:50:39.490346 (+ 144us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 22:50:39.490911 (+ 565us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 22:50:39.490919 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0929 22:50:39.492471 (+ 1552us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 22:50:39.494154 (+ 1683us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 22:50:39.494161 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 22:50:39.494164 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 22:50:39.494483 (+ 319us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 22:50:39.494681 (+ 198us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 22:50:39.494681 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 22:50:39.494682 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 22:50:39.494816 (+ 134us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 22:50:39.495091 (+ 275us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 22:50:39.495095 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 22:50:39.495221 (+ 126us) client_negotiation.cc:769] Sending connection context
0929 22:50:39.495306 (+ 85us) client_negotiation.cc:240] Negotiation successful
0929 22:50:39.495477 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":137,"thread_start_us":82,"threads_started":1}
I20260929 22:50:39.496228 21740 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.14.147.253" port: 40121 } has no permanent_uuid. Determining permanent_uuid...
I20260929 22:50:39.496901 21743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:50:39.488151 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59980 (local address 127.14.147.252:41485)
0929 22:50:39.488324 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0929 22:50:39.488328 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 22:50:39.488341 (+ 13us) server_negotiation.cc:407] Connection header received
0929 22:50:39.488358 (+ 17us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 22:50:39.488359 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 22:50:39.488376 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 22:50:39.488435 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0929 22:50:39.489018 (+ 583us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:50:39.489831 (+ 813us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:50:39.490732 (+ 901us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:50:39.491105 (+ 373us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:50:39.491163 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 22:50:39.493819 (+ 2656us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 22:50:39.493829 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 22:50:39.493830 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 22:50:39.493838 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0929 22:50:39.495624 (+ 1786us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 22:50:39.496094 (+ 470us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 22:50:39.496097 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 22:50:39.496097 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 22:50:39.496155 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 22:50:39.496365 (+ 210us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 22:50:39.496365 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 22:50:39.496366 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 22:50:39.496511 (+ 145us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 22:50:39.496638 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0929 22:50:39.496740 (+ 102us) server_negotiation.cc:299] Negotiation successful
0929 22:50:39.496823 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":148,"thread_start_us":106,"threads_started":1}
Sep 29 22:50:39 dist-test-slave-98mk krb5kdc[20968](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790722239, etypes {rep=17 tkt=17 ses=17}, oryx/127.14.147.252@KRBTEST.COM for oryx/127.14.147.253@KRBTEST.COM
I20260929 22:50:39.502535 21742 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:50:39.496482 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.147.253:40121 (local address 127.0.0.1:56782)
0929 22:50:39.496524 (+ 42us) negotiation.cc:107] Waiting for socket to connect
0929 22:50:39.496534 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0929 22:50:39.496564 (+ 30us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 22:50:39.496728 (+ 164us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 22:50:39.496729 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 22:50:39.496875 (+ 146us) client_negotiation.cc:189] Negotiated authn=SASL
0929 22:50:39.496934 (+ 59us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 22:50:39.496936 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 22:50:39.497692 (+ 756us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 22:50:39.497693 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 22:50:39.498200 (+ 507us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 22:50:39.498201 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 22:50:39.498535 (+ 334us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 22:50:39.498536 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 22:50:39.498591 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 22:50:39.498947 (+ 356us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 22:50:39.498952 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0929 22:50:39.500479 (+ 1527us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 22:50:39.501778 (+ 1299us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 22:50:39.501779 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 22:50:39.501779 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 22:50:39.501932 (+ 153us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 22:50:39.502083 (+ 151us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 22:50:39.502083 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 22:50:39.502083 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 22:50:39.502126 (+ 43us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 22:50:39.502303 (+ 177us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 22:50:39.502304 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 22:50:39.502341 (+ 37us) client_negotiation.cc:769] Sending connection context
0929 22:50:39.502386 (+ 45us) client_negotiation.cc:240] Negotiation successful
0929 22:50:39.502462 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":27}
I20260929 22:50:39.502542 21668 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:50:39.496569 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56782 (local address 127.14.147.253:40121)
0929 22:50:39.496615 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0929 22:50:39.496617 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 22:50:39.496624 (+ 7us) server_negotiation.cc:407] Connection header received
0929 22:50:39.496650 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 22:50:39.496651 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 22:50:39.496666 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 22:50:39.496719 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0929 22:50:39.497032 (+ 313us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:50:39.497620 (+ 588us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:50:39.498262 (+ 642us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:50:39.498477 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:50:39.498532 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 22:50:39.500573 (+ 2041us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 22:50:39.500574 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 22:50:39.500574 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 22:50:39.500581 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0929 22:50:39.501707 (+ 1126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 22:50:39.501992 (+ 285us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 22:50:39.501993 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 22:50:39.501993 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 22:50:39.502036 (+ 43us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 22:50:39.502164 (+ 128us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 22:50:39.502164 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 22:50:39.502164 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 22:50:39.502262 (+ 98us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 22:50:39.502307 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0929 22:50:39.502408 (+ 101us) server_negotiation.cc:299] Negotiation successful
0929 22:50:39.502477 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260929 22:50:39.502986 21740 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.14.147.252" port: 41485 } has no permanent_uuid. Determining permanent_uuid...
Sep 29 22:50:39 dist-test-slave-98mk krb5kdc[20968](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790722239, etypes {rep=17 tkt=17 ses=17}, oryx/127.14.147.252@KRBTEST.COM for oryx/127.14.147.252@KRBTEST.COM
I20260929 22:50:39.510355 21743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:50:39.503288 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59990 (local address 127.14.147.252:41485)
0929 22:50:39.503323 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0929 22:50:39.503325 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 22:50:39.503331 (+ 6us) server_negotiation.cc:407] Connection header received
0929 22:50:39.503386 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 22:50:39.503387 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 22:50:39.503395 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 22:50:39.503481 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0929 22:50:39.503689 (+ 208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:50:39.504152 (+ 463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:50:39.504740 (+ 588us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:50:39.504936 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:50:39.504981 (+ 45us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 22:50:39.507369 (+ 2388us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 22:50:39.507371 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 22:50:39.507371 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 22:50:39.507379 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0929 22:50:39.509142 (+ 1763us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 22:50:39.509556 (+ 414us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 22:50:39.509557 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 22:50:39.509557 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 22:50:39.509614 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 22:50:39.509787 (+ 173us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 22:50:39.509788 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 22:50:39.509788 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 22:50:39.509917 (+ 129us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 22:50:39.509977 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0929 22:50:39.510121 (+ 144us) server_negotiation.cc:299] Negotiation successful
0929 22:50:39.510236 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":19}
I20260929 22:50:39.510447 21742 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:50:39.503240 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.147.252:41485 (local address 127.0.0.1:59990)
0929 22:50:39.503285 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0929 22:50:39.503296 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0929 22:50:39.503338 (+ 42us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 22:50:39.503487 (+ 149us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 22:50:39.503488 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 22:50:39.503598 (+ 110us) client_negotiation.cc:189] Negotiated authn=SASL
0929 22:50:39.503632 (+ 34us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 22:50:39.503634 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 22:50:39.504212 (+ 578us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 22:50:39.504213 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 22:50:39.504686 (+ 473us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 22:50:39.504687 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 22:50:39.505102 (+ 415us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 22:50:39.505105 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 22:50:39.505200 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 22:50:39.505846 (+ 646us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 22:50:39.505853 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0929 22:50:39.507238 (+ 1385us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 22:50:39.509240 (+ 2002us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 22:50:39.509241 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 22:50:39.509242 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 22:50:39.509484 (+ 242us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 22:50:39.509669 (+ 185us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 22:50:39.509670 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 22:50:39.509670 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 22:50:39.509733 (+ 63us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 22:50:39.509975 (+ 242us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 22:50:39.509976 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 22:50:39.510027 (+ 51us) client_negotiation.cc:769] Sending connection context
0929 22:50:39.510095 (+ 68us) client_negotiation.cc:240] Negotiation successful
0929 22:50:39.510296 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":28,"mutex_wait_us":88}
I20260929 22:50:39.512749 21740 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 98a8b44ee91645f8a3dc6f02d1c5c698: Bootstrap starting.
I20260929 22:50:39.513593 21740 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 98a8b44ee91645f8a3dc6f02d1c5c698: Neither blocks nor log segments found. Creating new log.
I20260929 22:50:39.513911 21740 log.cc:824] T 00000000000000000000000000000000 P 98a8b44ee91645f8a3dc6f02d1c5c698: Log is configured to *not* fsync() on all Append() calls
I20260929 22:50:39.514803 21740 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 98a8b44ee91645f8a3dc6f02d1c5c698: No bootstrap required, opened a new log
I20260929 22:50:39.516739 21740 raft_consensus.cc:348] T 00000000000000000000000000000000 P 98a8b44ee91645f8a3dc6f02d1c5c698 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d38d0f6f0d3c4d2b8b7131e3f0f72ffe" member_type: VOTER last_known_addr { host: "127.14.147.254" port: 45601 } } peers { permanent_uuid: "e820b147283e4503b216af02b09dce4a" member_type: VOTER last_known_addr { host: "127.14.147.253" port: 40121 } } peers { permanent_uuid: "98a8b44ee91645f8a3dc6f02d1c5c698" member_type: VOTER last_known_addr { host: "127.14.147.252" port: 41485 } }
I20260929 22:50:39.516878 21740 raft_consensus.cc:374] T 00000000000000000000000000000000 P 98a8b44ee91645f8a3dc6f02d1c5c698 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 22:50:39.516901 21740 raft_consensus.cc:729] T 00000000000000000000000000000000 P 98a8b44ee91645f8a3dc6f02d1c5c698 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 98a8b44ee91645f8a3dc6f02d1c5c698, State: Initialized, Role: FOLLOWER
I20260929 22:50:39.516999 21740 consensus_queue.cc:260] T 00000000000000000000000000000000 P 98a8b44ee91645f8a3dc6f02d1c5c698 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d38d0f6f0d3c4d2b8b7131e3f0f72ffe" member_type: VOTER last_known_addr { host: "127.14.147.254" port: 45601 } } peers { permanent_uuid: "e820b147283e4503b216af02b09dce4a" member_type: VOTER last_known_addr { host: "127.14.147.253" port: 40121 } } peers { permanent_uuid: "98a8b44ee91645f8a3dc6f02d1c5c698" member_type: VOTER last_known_addr { host: "127.14.147.252" port: 41485 } }
I20260929 22:50:39.517393 21740 sys_catalog.cc:564] T 00000000000000000000000000000000 P 98a8b44ee91645f8a3dc6f02d1c5c698 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 22:50:39.517351 21745 sys_catalog.cc:455] T 00000000000000000000000000000000 P 98a8b44ee91645f8a3dc6f02d1c5c698 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d38d0f6f0d3c4d2b8b7131e3f0f72ffe" member_type: VOTER last_known_addr { host: "127.14.147.254" port: 45601 } } peers { permanent_uuid: "e820b147283e4503b216af02b09dce4a" member_type: VOTER last_known_addr { host: "127.14.147.253" port: 40121 } } peers { permanent_uuid: "98a8b44ee91645f8a3dc6f02d1c5c698" member_type: VOTER last_known_addr { host: "127.14.147.252" port: 41485 } } }
I20260929 22:50:39.517455 21745 sys_catalog.cc:458] T 00000000000000000000000000000000 P 98a8b44ee91645f8a3dc6f02d1c5c698 [sys.catalog]: This master's current role is: FOLLOWER
W20260929 22:50:39.519074 21756 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 98a8b44ee91645f8a3dc6f02d1c5c698: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 22:50:39.519165 21756 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260929 22:50:39.521281 21671 master_runner.cc:428] Detected that this master 98a8b44ee91645f8a3dc6f02d1c5c698 is joining an existing cluster
I20260929 22:50:39.521325 21671 master_runner.cc:432] Initiating AddMaster RPC to add 127.14.147.252:41485
I20260929 22:50:39.529058 21762 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:50:39.522243 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.147.253:40121 (local address 127.0.0.1:56798)
0929 22:50:39.522382 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0929 22:50:39.522394 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0929 22:50:39.522459 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 22:50:39.522609 (+ 150us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 22:50:39.522610 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 22:50:39.522829 (+ 219us) client_negotiation.cc:189] Negotiated authn=SASL
0929 22:50:39.522917 (+ 88us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 22:50:39.522919 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 22:50:39.523918 (+ 999us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 22:50:39.523919 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 22:50:39.524950 (+ 1031us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 22:50:39.524952 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 22:50:39.525454 (+ 502us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 22:50:39.525455 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 22:50:39.525508 (+ 53us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 22:50:39.525949 (+ 441us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 22:50:39.525953 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0929 22:50:39.526302 (+ 349us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 22:50:39.528088 (+ 1786us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 22:50:39.528089 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 22:50:39.528090 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 22:50:39.528259 (+ 169us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 22:50:39.528491 (+ 232us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 22:50:39.528492 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 22:50:39.528492 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 22:50:39.528547 (+ 55us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 22:50:39.528799 (+ 252us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 22:50:39.528799 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 22:50:39.528836 (+ 37us) client_negotiation.cc:769] Sending connection context
0929 22:50:39.528899 (+ 63us) client_negotiation.cc:240] Negotiation successful
0929 22:50:39.528982 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":110,"mutex_wait_us":37,"thread_start_us":84,"threads_started":1}
I20260929 22:50:39.529110 21668 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:50:39.522407 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56798 (local address 127.14.147.253:40121)
0929 22:50:39.522499 (+ 92us) server_negotiation.cc:206] Beginning negotiation
0929 22:50:39.522501 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 22:50:39.522509 (+ 8us) server_negotiation.cc:407] Connection header received
0929 22:50:39.522531 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 22:50:39.522532 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 22:50:39.522542 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 22:50:39.522588 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0929 22:50:39.523005 (+ 417us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:50:39.523830 (+ 825us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:50:39.525043 (+ 1213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:50:39.525342 (+ 299us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:50:39.525376 (+ 34us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 22:50:39.526400 (+ 1024us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 22:50:39.526401 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 22:50:39.526402 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 22:50:39.526410 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0929 22:50:39.527995 (+ 1585us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 22:50:39.528337 (+ 342us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 22:50:39.528338 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 22:50:39.528339 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 22:50:39.528402 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 22:50:39.528602 (+ 200us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 22:50:39.528604 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 22:50:39.528604 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 22:50:39.528739 (+ 135us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 22:50:39.528801 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0929 22:50:39.528936 (+ 135us) server_negotiation.cc:299] Negotiation successful
0929 22:50:39.529019 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260929 22:50:39.547390 21073 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:50:39.522511 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53134 (local address 127.14.147.254:45601)
0929 22:50:39.522555 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0929 22:50:39.522557 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 22:50:39.522793 (+ 236us) server_negotiation.cc:407] Connection header received
0929 22:50:39.522891 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 22:50:39.522892 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 22:50:39.522904 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 22:50:39.522936 (+ 32us) server_negotiation.cc:226] Negotiated authn=SASL
0929 22:50:39.523401 (+ 465us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:50:39.524173 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:50:39.525215 (+ 1042us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:50:39.525522 (+ 307us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:50:39.525564 (+ 42us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 22:50:39.544806 (+ 19242us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 22:50:39.544808 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 22:50:39.544809 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 22:50:39.544818 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0929 22:50:39.546160 (+ 1342us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 22:50:39.546621 (+ 461us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 22:50:39.546623 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 22:50:39.546623 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 22:50:39.546678 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 22:50:39.546840 (+ 162us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 22:50:39.546841 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 22:50:39.546841 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 22:50:39.546997 (+ 156us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 22:50:39.547053 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0929 22:50:39.547204 (+ 151us) server_negotiation.cc:299] Negotiation successful
0929 22:50:39.547295 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260929 22:50:39.548118 14927 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhWOCLk/build/release/bin/kudu with pid 21077
I20260929 22:50:39.548161 21743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:50:39.522444 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60004 (local address 127.14.147.252:41485)
0929 22:50:39.522640 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0929 22:50:39.522642 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 22:50:39.522770 (+ 128us) server_negotiation.cc:407] Connection header received
0929 22:50:39.522845 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 22:50:39.522845 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 22:50:39.522852 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 22:50:39.522875 (+ 23us) server_negotiation.cc:226] Negotiated authn=SASL
0929 22:50:39.523249 (+ 374us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:50:39.523948 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:50:39.525037 (+ 1089us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:50:39.525323 (+ 286us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:50:39.525359 (+ 36us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 22:50:39.544927 (+ 19568us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 22:50:39.544929 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 22:50:39.544929 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 22:50:39.544935 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0929 22:50:39.546556 (+ 1621us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 22:50:39.546899 (+ 343us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 22:50:39.546900 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 22:50:39.546900 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 22:50:39.546943 (+ 43us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 22:50:39.547112 (+ 169us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 22:50:39.547113 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 22:50:39.547113 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 22:50:39.547226 (+ 113us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 22:50:39.547284 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0929 22:50:39.547759 (+ 475us) server_negotiation.cc:299] Negotiation successful
0929 22:50:39.548041 (+ 282us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":206,"server-negotiator.queue_time_us":163}
I20260929 22:50:39.548872 21764 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:50:39.522253 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.147.252:41485 (local address 127.0.0.1:60004)
0929 22:50:39.522718 (+ 465us) negotiation.cc:107] Waiting for socket to connect
0929 22:50:39.522727 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0929 22:50:39.522789 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 22:50:39.522918 (+ 129us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 22:50:39.522919 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 22:50:39.523109 (+ 190us) client_negotiation.cc:189] Negotiated authn=SASL
0929 22:50:39.523174 (+ 65us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 22:50:39.523176 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 22:50:39.524019 (+ 843us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 22:50:39.524019 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 22:50:39.524950 (+ 931us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 22:50:39.524952 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 22:50:39.525445 (+ 493us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 22:50:39.525445 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 22:50:39.525501 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 22:50:39.544393 (+ 18892us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 22:50:39.544397 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0929 22:50:39.544836 (+ 439us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 22:50:39.546647 (+ 1811us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 22:50:39.546647 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 22:50:39.546648 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 22:50:39.546836 (+ 188us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 22:50:39.547003 (+ 167us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 22:50:39.547004 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 22:50:39.547004 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 22:50:39.547062 (+ 58us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 22:50:39.547281 (+ 219us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 22:50:39.547282 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 22:50:39.547337 (+ 55us) client_negotiation.cc:769] Sending connection context
0929 22:50:39.547403 (+ 66us) client_negotiation.cc:240] Negotiation successful
0929 22:50:39.548124 (+ 721us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":452,"mutex_wait_us":41,"spinlock_wait_cycles":9984,"thread_start_us":101,"threads_started":1}
I20260929 22:50:39.551594 21763 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:50:39.522607 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.147.254:45601 (local address 127.0.0.1:53134)
0929 22:50:39.522732 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0929 22:50:39.522741 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0929 22:50:39.522817 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 22:50:39.523001 (+ 184us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 22:50:39.523002 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 22:50:39.523229 (+ 227us) client_negotiation.cc:189] Negotiated authn=SASL
0929 22:50:39.523291 (+ 62us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 22:50:39.523293 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 22:50:39.524269 (+ 976us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 22:50:39.524270 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 22:50:39.525114 (+ 844us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 22:50:39.525117 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 22:50:39.525642 (+ 525us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 22:50:39.525643 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 22:50:39.525705 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 22:50:39.544089 (+ 18384us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 22:50:39.544096 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0929 22:50:39.544665 (+ 569us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 22:50:39.546245 (+ 1580us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 22:50:39.546247 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 22:50:39.546247 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 22:50:39.546555 (+ 308us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 22:50:39.546731 (+ 176us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 22:50:39.546732 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 22:50:39.546732 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 22:50:39.546790 (+ 58us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 22:50:39.547042 (+ 252us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 22:50:39.547042 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 22:50:39.547088 (+ 46us) client_negotiation.cc:769] Sending connection context
0929 22:50:39.547858 (+ 770us) client_negotiation.cc:240] Negotiation successful
0929 22:50:39.548108 (+ 250us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":110,"mutex_wait_us":150,"thread_start_us":74,"threads_started":1}
I20260929 22:50:39.553493 14927 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhWOCLk/build/release/bin/kudu
/tmp/dist-test-taskhWOCLk/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790722213368738-14927-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790722213368738-14927-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790722213368738-14927-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790722213368738-14927-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.147.193:44967
--local_ip_for_outbound_sockets=127.14.147.193
--tserver_master_addrs=127.14.147.254:45601
--webserver_port=41813
--webserver_interface=127.14.147.193
--keytab_file=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790722213368738-14927-0/krb5kdc/oryx_127.14.147.193.keytab
--principal=oryx/127.14.147.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.147.212:34343
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.14.147.254:45601,127.14.147.253:40121
--tserver_master_addrs=127.14.147.254:45601,127.14.147.253:40121,127.14.147.252:41485 with env {KRB5CCNAME=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790722213368738-14927-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790722213368738-14927-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790722213368738-14927-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 29 22:50:39 dist-test-slave-98mk krb5kdc[20968](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790722238, etypes {rep=17 tkt=17 ses=17}, oryx/127.14.147.254@KRBTEST.COM for oryx/127.14.147.252@KRBTEST.COM
I20260929 22:50:39.562457 21576 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:50:39.554260 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.147.252:41485 (local address 127.0.0.1:60012)
0929 22:50:39.554321 (+ 61us) negotiation.cc:107] Waiting for socket to connect
0929 22:50:39.554331 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0929 22:50:39.554367 (+ 36us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 22:50:39.554500 (+ 133us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 22:50:39.554501 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 22:50:39.554684 (+ 183us) client_negotiation.cc:189] Negotiated authn=SASL
0929 22:50:39.554737 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 22:50:39.554739 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 22:50:39.555688 (+ 949us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 22:50:39.555690 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 22:50:39.556527 (+ 837us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 22:50:39.556531 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 22:50:39.557018 (+ 487us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 22:50:39.557019 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 22:50:39.557083 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 22:50:39.557511 (+ 428us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 22:50:39.557515 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0929 22:50:39.559578 (+ 2063us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 22:50:39.561378 (+ 1800us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 22:50:39.561379 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 22:50:39.561380 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 22:50:39.561598 (+ 218us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 22:50:39.561790 (+ 192us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 22:50:39.561790 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 22:50:39.561791 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 22:50:39.561847 (+ 56us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 22:50:39.562114 (+ 267us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 22:50:39.562114 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 22:50:39.562163 (+ 49us) client_negotiation.cc:769] Sending connection context
0929 22:50:39.562222 (+ 59us) client_negotiation.cc:240] Negotiation successful
0929 22:50:39.562350 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":41}
I20260929 22:50:39.562850 21743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:50:39.554316 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60012 (local address 127.14.147.252:41485)
0929 22:50:39.554352 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0929 22:50:39.554354 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 22:50:39.554370 (+ 16us) server_negotiation.cc:407] Connection header received
0929 22:50:39.554431 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 22:50:39.554431 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 22:50:39.554441 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 22:50:39.554496 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0929 22:50:39.554822 (+ 326us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:50:39.555596 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:50:39.556620 (+ 1024us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:50:39.556910 (+ 290us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:50:39.556964 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 22:50:39.559685 (+ 2721us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 22:50:39.559686 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 22:50:39.559687 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 22:50:39.559694 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0929 22:50:39.561277 (+ 1583us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 22:50:39.561672 (+ 395us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 22:50:39.561673 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 22:50:39.561674 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 22:50:39.561727 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 22:50:39.561904 (+ 177us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 22:50:39.561904 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 22:50:39.561905 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 22:50:39.562038 (+ 133us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 22:50:39.562090 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0929 22:50:39.562683 (+ 593us) server_negotiation.cc:299] Negotiation successful
0929 22:50:39.562763 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260929 22:50:39.563350 21014 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.14.147.252:41485: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "98a8b44ee91645f8a3dc6f02d1c5c698"
member_type: NON_VOTER
last_known_addr {
host: "127.14.147.252"
port: 41485
}
attrs {
promote: true
}
}
dest_uuid: "d38d0f6f0d3c4d2b8b7131e3f0f72ffe"
cas_config_opid_index: 5
I20260929 22:50:39.563661 21014 consensus_queue.cc:237] T 00000000000000000000000000000000 P d38d0f6f0d3c4d2b8b7131e3f0f72ffe [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "d38d0f6f0d3c4d2b8b7131e3f0f72ffe" member_type: VOTER last_known_addr { host: "127.14.147.254" port: 45601 } } peers { permanent_uuid: "e820b147283e4503b216af02b09dce4a" member_type: NON_VOTER last_known_addr { host: "127.14.147.253" port: 40121 } attrs { promote: true } } peers { permanent_uuid: "98a8b44ee91645f8a3dc6f02d1c5c698" member_type: NON_VOTER last_known_addr { host: "127.14.147.252" port: 41485 } attrs { promote: true } }
I20260929 22:50:39.564018 21577 raft_consensus.cc:2932] T 00000000000000000000000000000000 P d38d0f6f0d3c4d2b8b7131e3f0f72ffe [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 98a8b44ee91645f8a3dc6f02d1c5c698 (127.14.147.252) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "d38d0f6f0d3c4d2b8b7131e3f0f72ffe" member_type: VOTER last_known_addr { host: "127.14.147.254" port: 45601 } } peers { permanent_uuid: "e820b147283e4503b216af02b09dce4a" member_type: NON_VOTER last_known_addr { host: "127.14.147.253" port: 40121 } attrs { promote: true } } peers { permanent_uuid: "98a8b44ee91645f8a3dc6f02d1c5c698" member_type: NON_VOTER last_known_addr { host: "127.14.147.252" port: 41485 } attrs { promote: true } } }
I20260929 22:50:39.564499 21713 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 98a8b44ee91645f8a3dc6f02d1c5c698 [term 0 FOLLOWER]: Advancing to term 1
I20260929 22:50:39.565035 21577 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.14.147.252:41485
I20260929 22:50:39.565105 21577 sys_catalog.cc:455] T 00000000000000000000000000000000 P d38d0f6f0d3c4d2b8b7131e3f0f72ffe [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "d38d0f6f0d3c4d2b8b7131e3f0f72ffe" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "d38d0f6f0d3c4d2b8b7131e3f0f72ffe" member_type: VOTER last_known_addr { host: "127.14.147.254" port: 45601 } } peers { permanent_uuid: "e820b147283e4503b216af02b09dce4a" member_type: NON_VOTER last_known_addr { host: "127.14.147.253" port: 40121 } attrs { promote: true } } peers { permanent_uuid: "98a8b44ee91645f8a3dc6f02d1c5c698" member_type: NON_VOTER last_known_addr { host: "127.14.147.252" port: 41485 } attrs { promote: true } } }
I20260929 22:50:39.565164 21577 sys_catalog.cc:458] T 00000000000000000000000000000000 P d38d0f6f0d3c4d2b8b7131e3f0f72ffe [sys.catalog]: This master's current role is: LEADER
I20260929 22:50:39.565428 21671 master.cc:537] Master@127.14.147.252:41485 shutting down...
I20260929 22:50:39.565436 21713 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 98a8b44ee91645f8a3dc6f02d1c5c698 [term 1 FOLLOWER]: Refusing update from remote peer d38d0f6f0d3c4d2b8b7131e3f0f72ffe: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260929 22:50:39.565979 21578 consensus_queue.cc:1035] T 00000000000000000000000000000000 P d38d0f6f0d3c4d2b8b7131e3f0f72ffe [LEADER]: Connected to new peer: Peer: permanent_uuid: "98a8b44ee91645f8a3dc6f02d1c5c698" member_type: NON_VOTER last_known_addr { host: "127.14.147.252" port: 41485 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 0, Time since last communication: 0.000s
W20260929 22:50:39.566784 21000 consensus_peers.cc:487] T 00000000000000000000000000000000 P d38d0f6f0d3c4d2b8b7131e3f0f72ffe -> Peer 98a8b44ee91645f8a3dc6f02d1c5c698 (127.14.147.252:41485): Couldn't send request to peer 98a8b44ee91645f8a3dc6f02d1c5c698. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260929 22:50:39.571313 21671 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 98a8b44ee91645f8a3dc6f02d1c5c698 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 22:50:39.571449 21671 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 98a8b44ee91645f8a3dc6f02d1c5c698 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 22:50:39.571496 21671 tablet_replica.cc:330] T 00000000000000000000000000000000 P 98a8b44ee91645f8a3dc6f02d1c5c698: stopping tablet replica
I20260929 22:50:39.574218 21576 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:50:39.564081 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.147.253:40121 (local address 127.0.0.1:56804)
0929 22:50:39.566968 (+ 2887us) negotiation.cc:107] Waiting for socket to connect
0929 22:50:39.566979 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0929 22:50:39.567052 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 22:50:39.570539 (+ 3487us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 22:50:39.570541 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 22:50:39.570543 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 22:50:39.570598 (+ 55us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 22:50:39.570600 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 22:50:39.571750 (+ 1150us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 22:50:39.571751 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 22:50:39.572977 (+ 1226us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 22:50:39.572982 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 22:50:39.573956 (+ 974us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 22:50:39.573957 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 22:50:39.574024 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 22:50:39.574049 (+ 25us) client_negotiation.cc:769] Sending connection context
0929 22:50:39.574099 (+ 50us) client_negotiation.cc:240] Negotiation successful
0929 22:50:39.574143 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2869}
I20260929 22:50:39.574260 21668 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:50:39.564177 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56804 (local address 127.14.147.253:40121)
0929 22:50:39.565678 (+ 1501us) server_negotiation.cc:206] Beginning negotiation
0929 22:50:39.565680 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 22:50:39.567023 (+ 1343us) server_negotiation.cc:407] Connection header received
0929 22:50:39.567124 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 22:50:39.567125 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 22:50:39.567148 (+ 23us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 22:50:39.567184 (+ 36us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 22:50:39.570686 (+ 3502us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:50:39.571656 (+ 970us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:50:39.573083 (+ 1427us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:50:39.573696 (+ 613us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:50:39.573743 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 22:50:39.573811 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0929 22:50:39.574126 (+ 315us) server_negotiation.cc:299] Negotiation successful
0929 22:50:39.574188 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1472}
I20260929 22:50:39.574810 21627 raft_consensus.cc:1261] T 00000000000000000000000000000000 P e820b147283e4503b216af02b09dce4a [term 1 LEARNER]: Refusing update from remote peer d38d0f6f0d3c4d2b8b7131e3f0f72ffe: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260929 22:50:39.575026 21578 consensus_queue.cc:1035] T 00000000000000000000000000000000 P d38d0f6f0d3c4d2b8b7131e3f0f72ffe [LEADER]: Connected to new peer: Peer: permanent_uuid: "e820b147283e4503b216af02b09dce4a" member_type: NON_VOTER last_known_addr { host: "127.14.147.253" port: 40121 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260929 22:50:39.575634 21627 raft_consensus.cc:2932] T 00000000000000000000000000000000 P e820b147283e4503b216af02b09dce4a [term 1 LEARNER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 98a8b44ee91645f8a3dc6f02d1c5c698 (127.14.147.252) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "d38d0f6f0d3c4d2b8b7131e3f0f72ffe" member_type: VOTER last_known_addr { host: "127.14.147.254" port: 45601 } } peers { permanent_uuid: "e820b147283e4503b216af02b09dce4a" member_type: NON_VOTER last_known_addr { host: "127.14.147.253" port: 40121 } attrs { promote: true } } peers { permanent_uuid: "98a8b44ee91645f8a3dc6f02d1c5c698" member_type: NON_VOTER last_known_addr { host: "127.14.147.252" port: 41485 } attrs { promote: true } } }
I20260929 22:50:39.576960 21767 sys_catalog.cc:455] T 00000000000000000000000000000000 P e820b147283e4503b216af02b09dce4a [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "d38d0f6f0d3c4d2b8b7131e3f0f72ffe" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "d38d0f6f0d3c4d2b8b7131e3f0f72ffe" member_type: VOTER last_known_addr { host: "127.14.147.254" port: 45601 } } peers { permanent_uuid: "e820b147283e4503b216af02b09dce4a" member_type: NON_VOTER last_known_addr { host: "127.14.147.253" port: 40121 } attrs { promote: true } } peers { permanent_uuid: "98a8b44ee91645f8a3dc6f02d1c5c698" member_type: NON_VOTER last_known_addr { host: "127.14.147.252" port: 41485 } attrs { promote: true } } }
I20260929 22:50:39.577059 21767 sys_catalog.cc:458] T 00000000000000000000000000000000 P e820b147283e4503b216af02b09dce4a [sys.catalog]: This master's current role is: LEARNER
I20260929 22:50:39.577088 21656 sys_catalog.cc:455] T 00000000000000000000000000000000 P e820b147283e4503b216af02b09dce4a [sys.catalog]: SysCatalogTable state changed. Reason: New leader d38d0f6f0d3c4d2b8b7131e3f0f72ffe. Latest consensus state: current_term: 1 leader_uuid: "d38d0f6f0d3c4d2b8b7131e3f0f72ffe" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "d38d0f6f0d3c4d2b8b7131e3f0f72ffe" member_type: VOTER last_known_addr { host: "127.14.147.254" port: 45601 } } peers { permanent_uuid: "e820b147283e4503b216af02b09dce4a" member_type: NON_VOTER last_known_addr { host: "127.14.147.253" port: 40121 } attrs { promote: true } } peers { permanent_uuid: "98a8b44ee91645f8a3dc6f02d1c5c698" member_type: NON_VOTER last_known_addr { host: "127.14.147.252" port: 41485 } attrs { promote: true } } }
I20260929 22:50:39.577178 21656 sys_catalog.cc:458] T 00000000000000000000000000000000 P e820b147283e4503b216af02b09dce4a [sys.catalog]: This master's current role is: LEARNER
I20260929 22:50:39.577450 21577 raft_consensus.cc:1050] T 00000000000000000000000000000000 P d38d0f6f0d3c4d2b8b7131e3f0f72ffe: attempting to promote NON_VOTER e820b147283e4503b216af02b09dce4a to VOTER
I20260929 22:50:39.577654 21577 consensus_queue.cc:237] T 00000000000000000000000000000000 P d38d0f6f0d3c4d2b8b7131e3f0f72ffe [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "d38d0f6f0d3c4d2b8b7131e3f0f72ffe" member_type: VOTER last_known_addr { host: "127.14.147.254" port: 45601 } } peers { permanent_uuid: "e820b147283e4503b216af02b09dce4a" member_type: VOTER last_known_addr { host: "127.14.147.253" port: 40121 } attrs { promote: false } } peers { permanent_uuid: "98a8b44ee91645f8a3dc6f02d1c5c698" member_type: NON_VOTER last_known_addr { host: "127.14.147.252" port: 41485 } attrs { promote: true } }
I20260929 22:50:39.578362 21627 raft_consensus.cc:1261] T 00000000000000000000000000000000 P e820b147283e4503b216af02b09dce4a [term 1 LEARNER]: Refusing update from remote peer d38d0f6f0d3c4d2b8b7131e3f0f72ffe: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20260929 22:50:39.578369 21576 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:50:39.578224 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.147.252:41485 (local address 127.0.0.1:60024)
0929 22:50:39.578281 (+ 57us) negotiation.cc:107] Waiting for socket to connect
0929 22:50:39.578338 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.147.252:41485: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
I20260929 22:50:39.578533 21578 consensus_queue.cc:1035] T 00000000000000000000000000000000 P d38d0f6f0d3c4d2b8b7131e3f0f72ffe [LEADER]: Connected to new peer: Peer: permanent_uuid: "e820b147283e4503b216af02b09dce4a" member_type: VOTER last_known_addr { host: "127.14.147.253" port: 40121 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260929 22:50:39.579226 21576 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:50:39.579035 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.147.252:41485 (local address 127.0.0.1:60040)
0929 22:50:39.579114 (+ 79us) negotiation.cc:107] Waiting for socket to connect
0929 22:50:39.579142 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.147.252:41485: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65}
I20260929 22:50:39.579167 21578 raft_consensus.cc:2932] T 00000000000000000000000000000000 P d38d0f6f0d3c4d2b8b7131e3f0f72ffe [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, e820b147283e4503b216af02b09dce4a (127.14.147.253) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "d38d0f6f0d3c4d2b8b7131e3f0f72ffe" member_type: VOTER last_known_addr { host: "127.14.147.254" port: 45601 } } peers { permanent_uuid: "e820b147283e4503b216af02b09dce4a" member_type: VOTER last_known_addr { host: "127.14.147.253" port: 40121 } attrs { promote: false } } peers { permanent_uuid: "98a8b44ee91645f8a3dc6f02d1c5c698" member_type: NON_VOTER last_known_addr { host: "127.14.147.252" port: 41485 } attrs { promote: true } } }
W20260929 22:50:39.579371 21000 consensus_peers.cc:487] T 00000000000000000000000000000000 P d38d0f6f0d3c4d2b8b7131e3f0f72ffe -> Peer 98a8b44ee91645f8a3dc6f02d1c5c698 (127.14.147.252:41485): Couldn't send request to peer 98a8b44ee91645f8a3dc6f02d1c5c698. Status: Network error: Client connection negotiation failed: client connection to 127.14.147.252:41485: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260929 22:50:39.580217 21578 sys_catalog.cc:455] T 00000000000000000000000000000000 P d38d0f6f0d3c4d2b8b7131e3f0f72ffe [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "d38d0f6f0d3c4d2b8b7131e3f0f72ffe" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "d38d0f6f0d3c4d2b8b7131e3f0f72ffe" member_type: VOTER last_known_addr { host: "127.14.147.254" port: 45601 } } peers { permanent_uuid: "e820b147283e4503b216af02b09dce4a" member_type: VOTER last_known_addr { host: "127.14.147.253" port: 40121 } attrs { promote: false } } peers { permanent_uuid: "98a8b44ee91645f8a3dc6f02d1c5c698" member_type: NON_VOTER last_known_addr { host: "127.14.147.252" port: 41485 } attrs { promote: true } } }
I20260929 22:50:39.580298 21578 sys_catalog.cc:458] T 00000000000000000000000000000000 P d38d0f6f0d3c4d2b8b7131e3f0f72ffe [sys.catalog]: This master's current role is: LEADER
I20260929 22:50:39.580508 21577 sys_catalog.cc:455] T 00000000000000000000000000000000 P d38d0f6f0d3c4d2b8b7131e3f0f72ffe [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "d38d0f6f0d3c4d2b8b7131e3f0f72ffe" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "d38d0f6f0d3c4d2b8b7131e3f0f72ffe" member_type: VOTER last_known_addr { host: "127.14.147.254" port: 45601 } } peers { permanent_uuid: "e820b147283e4503b216af02b09dce4a" member_type: VOTER last_known_addr { host: "127.14.147.253" port: 40121 } attrs { promote: false } } peers { permanent_uuid: "98a8b44ee91645f8a3dc6f02d1c5c698" member_type: NON_VOTER last_known_addr { host: "127.14.147.252" port: 41485 } attrs { promote: true } } }
I20260929 22:50:39.580574 21577 sys_catalog.cc:458] T 00000000000000000000000000000000 P d38d0f6f0d3c4d2b8b7131e3f0f72ffe [sys.catalog]: This master's current role is: LEADER
I20260929 22:50:39.580821 21627 raft_consensus.cc:2932] T 00000000000000000000000000000000 P e820b147283e4503b216af02b09dce4a [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, e820b147283e4503b216af02b09dce4a (127.14.147.253) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "d38d0f6f0d3c4d2b8b7131e3f0f72ffe" member_type: VOTER last_known_addr { host: "127.14.147.254" port: 45601 } } peers { permanent_uuid: "e820b147283e4503b216af02b09dce4a" member_type: VOTER last_known_addr { host: "127.14.147.253" port: 40121 } attrs { promote: false } } peers { permanent_uuid: "98a8b44ee91645f8a3dc6f02d1c5c698" member_type: NON_VOTER last_known_addr { host: "127.14.147.252" port: 41485 } attrs { promote: true } } }
I20260929 22:50:39.581698 21656 sys_catalog.cc:455] T 00000000000000000000000000000000 P e820b147283e4503b216af02b09dce4a [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "d38d0f6f0d3c4d2b8b7131e3f0f72ffe" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "d38d0f6f0d3c4d2b8b7131e3f0f72ffe" member_type: VOTER last_known_addr { host: "127.14.147.254" port: 45601 } } peers { permanent_uuid: "e820b147283e4503b216af02b09dce4a" member_type: VOTER last_known_addr { host: "127.14.147.253" port: 40121 } attrs { promote: false } } peers { permanent_uuid: "98a8b44ee91645f8a3dc6f02d1c5c698" member_type: NON_VOTER last_known_addr { host: "127.14.147.252" port: 41485 } attrs { promote: true } } }
I20260929 22:50:39.581780 21656 sys_catalog.cc:458] T 00000000000000000000000000000000 P e820b147283e4503b216af02b09dce4a [sys.catalog]: This master's current role is: FOLLOWER
I20260929 22:50:39.584537 21671 master.cc:552] Master@127.14.147.252:41485 shutdown complete.
I20260929 22:50:39.584585 21671 master_runner.cc:305] Clearing existing system tablet
I20260929 22:50:39.584703 21671 fs_manager.cc:359] Metadata directory not provided
I20260929 22:50:39.584746 21671 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 22:50:39.585415 21671 fs_manager.cc:709] Time spent opening directory manager: real 0.000s user 0.001s sys 0.000s
I20260929 22:50:39.585876 21771 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 22:50:39.586046 21671 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790722213368738-14927-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 22:50:39.586102 21671 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260929 22:50:39.586124 21671 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790722213368738-14927-0/minicluster-data/master-2/data,/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790722213368738-14927-0/minicluster-data/master-2/wal
uuid: "98a8b44ee91645f8a3dc6f02d1c5c698"
format_stamp: "Formatted at 2026-09-29 22:50:39 on dist-test-slave-98mk"
server_key: "887e4c07ce432cbb9d939f466c01d244"
server_key_iv: "82d69a50576c0fe675c6948659dd547c"
server_key_version: "encryptionkey@0"
I20260929 22:50:39.586797 21671 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P 98a8b44ee91645f8a3dc6f02d1c5c698: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 22:50:39.588251 21671 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P 98a8b44ee91645f8a3dc6f02d1c5c698: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260929 22:50:39.588320 21671 log.cc:1196] T 00000000000000000000000000000000 P 98a8b44ee91645f8a3dc6f02d1c5c698: Deleting WAL directory at /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790722213368738-14927-0/minicluster-data/master-2/wal/wals/00000000000000000000000000000000
I20260929 22:50:39.588497 21671 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P 98a8b44ee91645f8a3dc6f02d1c5c698: Deleting consensus metadata
I20260929 22:50:39.588708 21671 master_runner.cc:315] Copying system tablet from 127.14.147.254:45601
I20260929 22:50:39.589313 21671 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P 98a8b44ee91645f8a3dc6f02d1c5c698: tablet copy: Beginning tablet copy session from remote peer at address 127.14.147.254:45601
I20260929 22:50:39.596361 21073 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:50:39.589773 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53138 (local address 127.14.147.254:45601)
0929 22:50:39.589821 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0929 22:50:39.589823 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 22:50:39.589832 (+ 9us) server_negotiation.cc:407] Connection header received
0929 22:50:39.589860 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 22:50:39.589861 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 22:50:39.589872 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 22:50:39.589935 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0929 22:50:39.590291 (+ 356us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:50:39.591029 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:50:39.592067 (+ 1038us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:50:39.592370 (+ 303us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:50:39.592448 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 22:50:39.593683 (+ 1235us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 22:50:39.593685 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 22:50:39.593686 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 22:50:39.593693 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0929 22:50:39.595200 (+ 1507us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 22:50:39.595606 (+ 406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 22:50:39.595606 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 22:50:39.595607 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 22:50:39.595668 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 22:50:39.595847 (+ 179us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 22:50:39.595847 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 22:50:39.595848 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 22:50:39.595980 (+ 132us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 22:50:39.596043 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0929 22:50:39.596186 (+ 143us) server_negotiation.cc:299] Negotiation successful
0929 22:50:39.596265 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260929 22:50:39.596367 21777 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:50:39.589597 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.147.254:45601 (local address 127.0.0.1:53138)
0929 22:50:39.589728 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0929 22:50:39.589741 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 22:50:39.589801 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 22:50:39.589946 (+ 145us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 22:50:39.589947 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 22:50:39.590137 (+ 190us) client_negotiation.cc:189] Negotiated authn=SASL
0929 22:50:39.590206 (+ 69us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 22:50:39.590209 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 22:50:39.591118 (+ 909us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 22:50:39.591119 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 22:50:39.591973 (+ 854us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 22:50:39.591975 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 22:50:39.592463 (+ 488us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 22:50:39.592464 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 22:50:39.592521 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 22:50:39.593188 (+ 667us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 22:50:39.593193 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0929 22:50:39.593591 (+ 398us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 22:50:39.595282 (+ 1691us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 22:50:39.595284 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 22:50:39.595284 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 22:50:39.595540 (+ 256us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 22:50:39.595732 (+ 192us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 22:50:39.595732 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 22:50:39.595733 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 22:50:39.595792 (+ 59us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 22:50:39.596042 (+ 250us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 22:50:39.596042 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 22:50:39.596092 (+ 50us) client_negotiation.cc:769] Sending connection context
0929 22:50:39.596157 (+ 65us) client_negotiation.cc:240] Negotiation successful
0929 22:50:39.596265 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":106,"thread_start_us":74,"threads_started":1}
I20260929 22:50:39.596933 21034 tablet_copy_service.cc:140] P d38d0f6f0d3c4d2b8b7131e3f0f72ffe: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 98a8b44ee91645f8a3dc6f02d1c5c698 ({username='oryx', principal='oryx/127.14.147.252@KRBTEST.COM'} at 127.0.0.1:53138)
I20260929 22:50:39.597000 21034 tablet_copy_service.cc:161] P d38d0f6f0d3c4d2b8b7131e3f0f72ffe: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 98a8b44ee91645f8a3dc6f02d1c5c698 at {username='oryx', principal='oryx/127.14.147.252@KRBTEST.COM'} at 127.0.0.1:53138: session id = 98a8b44ee91645f8a3dc6f02d1c5c698-00000000000000000000000000000000
I20260929 22:50:39.597570 21034 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P d38d0f6f0d3c4d2b8b7131e3f0f72ffe: Tablet Copy: opened 0 blocks and 1 log segments
I20260929 22:50:39.598098 21671 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260929 22:50:39.599459 21671 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P 98a8b44ee91645f8a3dc6f02d1c5c698: tablet copy: Starting download of 0 data blocks...
I20260929 22:50:39.599601 21671 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P 98a8b44ee91645f8a3dc6f02d1c5c698: tablet copy: Starting download of 1 WAL segments...
I20260929 22:50:39.600399 21671 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P 98a8b44ee91645f8a3dc6f02d1c5c698: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260929 22:50:39.601346 21034 tablet_copy_service.cc:342] P d38d0f6f0d3c4d2b8b7131e3f0f72ffe: Request end of tablet copy session 98a8b44ee91645f8a3dc6f02d1c5c698-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.14.147.252@KRBTEST.COM'} at 127.0.0.1:53138
I20260929 22:50:39.601403 21034 tablet_copy_service.cc:433] P d38d0f6f0d3c4d2b8b7131e3f0f72ffe: ending tablet copy session 98a8b44ee91645f8a3dc6f02d1c5c698-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 98a8b44ee91645f8a3dc6f02d1c5c698
I20260929 22:50:39.604020 21671 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 22:50:39.605010 21778 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 22:50:39.608341 21779 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 22:50:39.608529 21671 server_base.cc:1034] running on GCE node
W20260929 22:50:39.608551 21781 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 22:50:39.608852 21671 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 22:50:39.608999 21671 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 22:50:39.610121 21671 hybrid_clock.cc:648] HybridClock initialized: now 1790722239610127 us; error 47 us; skew 500 ppm
Sep 29 22:50:39 dist-test-slave-98mk krb5kdc[20968](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790722239, etypes {rep=17 tkt=17 ses=17}, oryx/127.14.147.252@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 22:50:39.611649 21671 init.cc:377] Logged in from keytab as oryx/127.14.147.252@KRBTEST.COM (short username oryx)
I20260929 22:50:39.611927 21671 webserver.cc:460] Webserver started at http://127.14.147.252:39503/ using document root <none> and password file <none>
I20260929 22:50:39.612035 21671 fs_manager.cc:359] Metadata directory not provided
I20260929 22:50:39.612066 21671 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 22:50:39.612757 21671 fs_manager.cc:709] Time spent opening directory manager: real 0.000s user 0.001s sys 0.000s
I20260929 22:50:39.613293 21788 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 22:50:39.613456 21671 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260929 22:50:39.613502 21671 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790722213368738-14927-0/minicluster-data/master-2/data,/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790722213368738-14927-0/minicluster-data/master-2/wal
uuid: "98a8b44ee91645f8a3dc6f02d1c5c698"
format_stamp: "Formatted at 2026-09-29 22:50:39 on dist-test-slave-98mk"
server_key: "887e4c07ce432cbb9d939f466c01d244"
server_key_iv: "82d69a50576c0fe675c6948659dd547c"
server_key_version: "encryptionkey@0"
I20260929 22:50:39.613559 21671 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790722213368738-14927-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790722213368738-14927-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790722213368738-14927-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 22:50:39.626346 21671 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 22:50:39.628137 21671 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 22:50:39.631073 21671 rpc_server.cc:307] RPC server started. Bound to: 127.14.147.252:41485
I20260929 22:50:39.631177 21840 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.147.252:41485 every 8 connection(s)
I20260929 22:50:39.631402 21671 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790722213368738-14927-0/minicluster-data/master-2/data/info.pb
I20260929 22:50:39.632308 21841 sys_catalog.cc:263] Verifying existing consensus state
I20260929 22:50:39.632838 21841 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 98a8b44ee91645f8a3dc6f02d1c5c698: Bootstrap starting.
I20260929 22:50:39.635828 21841 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 98a8b44ee91645f8a3dc6f02d1c5c698: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260929 22:50:39.636044 21841 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 98a8b44ee91645f8a3dc6f02d1c5c698: Bootstrap complete.
I20260929 22:50:39.636255 21841 raft_consensus.cc:348] T 00000000000000000000000000000000 P 98a8b44ee91645f8a3dc6f02d1c5c698 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "d38d0f6f0d3c4d2b8b7131e3f0f72ffe" member_type: VOTER last_known_addr { host: "127.14.147.254" port: 45601 } } peers { permanent_uuid: "e820b147283e4503b216af02b09dce4a" member_type: VOTER last_known_addr { host: "127.14.147.253" port: 40121 } attrs { promote: false } } peers { permanent_uuid: "98a8b44ee91645f8a3dc6f02d1c5c698" member_type: NON_VOTER last_known_addr { host: "127.14.147.252" port: 41485 } attrs { promote: true } }
I20260929 22:50:39.636354 21841 raft_consensus.cc:729] T 00000000000000000000000000000000 P 98a8b44ee91645f8a3dc6f02d1c5c698 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 98a8b44ee91645f8a3dc6f02d1c5c698, State: Initialized, Role: LEARNER
I20260929 22:50:39.636386 21841 consensus_queue.cc:260] T 00000000000000000000000000000000 P 98a8b44ee91645f8a3dc6f02d1c5c698 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "d38d0f6f0d3c4d2b8b7131e3f0f72ffe" member_type: VOTER last_known_addr { host: "127.14.147.254" port: 45601 } } peers { permanent_uuid: "e820b147283e4503b216af02b09dce4a" member_type: VOTER last_known_addr { host: "127.14.147.253" port: 40121 } attrs { promote: false } } peers { permanent_uuid: "98a8b44ee91645f8a3dc6f02d1c5c698" member_type: NON_VOTER last_known_addr { host: "127.14.147.252" port: 41485 } attrs { promote: true } }
I20260929 22:50:39.636586 21841 sys_catalog.cc:564] T 00000000000000000000000000000000 P 98a8b44ee91645f8a3dc6f02d1c5c698 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 22:50:39.637923 21844 sys_catalog.cc:455] T 00000000000000000000000000000000 P 98a8b44ee91645f8a3dc6f02d1c5c698 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "d38d0f6f0d3c4d2b8b7131e3f0f72ffe" member_type: VOTER last_known_addr { host: "127.14.147.254" port: 45601 } } peers { permanent_uuid: "e820b147283e4503b216af02b09dce4a" member_type: VOTER last_known_addr { host: "127.14.147.253" port: 40121 } attrs { promote: false } } peers { permanent_uuid: "98a8b44ee91645f8a3dc6f02d1c5c698" member_type: NON_VOTER last_known_addr { host: "127.14.147.252" port: 41485 } attrs { promote: true } } }
I20260929 22:50:39.638024 21844 sys_catalog.cc:458] T 00000000000000000000000000000000 P 98a8b44ee91645f8a3dc6f02d1c5c698 [sys.catalog]: This master's current role is: LEARNER
I20260929 22:50:39.639144 21855 catalog_manager.cc:1259] Loaded cluster ID: eeb78789c1ae4d64a674c24edf4ad136
I20260929 22:50:39.639194 21855 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 98a8b44ee91645f8a3dc6f02d1c5c698: loading cluster ID for follower catalog manager: success
I20260929 22:50:39.640239 21855 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 98a8b44ee91645f8a3dc6f02d1c5c698: acquiring CA information for follower catalog manager: success
I20260929 22:50:39.640545 21855 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 98a8b44ee91645f8a3dc6f02d1c5c698: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260929 22:50:39.653784 21765 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 22:50:39.653992 21765 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 22:50:39.654017 21765 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 22:50:39.655247 21765 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 22:50:39.655290 21765 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 22:50:39.655318 21765 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.147.193
I20260929 22:50:39.656939 21765 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.147.212:34343
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790722213368738-14927-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790722213368738-14927-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790722213368738-14927-0/krb5kdc/oryx_127.14.147.193.keytab
--principal=oryx/127.14.147.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.147.193:44967
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790722213368738-14927-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.14.147.193
--webserver_port=41813
--tserver_master_addrs=127.14.147.254:45601,127.14.147.253:40121,127.14.147.252:41485
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21765
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.147.193
--log_dir=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790722213368738-14927-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 21:58:54 UTC on ccycloud.quasar-xguoti.root.comops.site
build id 4681390
I20260929 22:50:39.657174 21765 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 22:50:39.657400 21765 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 22:50:39.662360 21861 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 22:50:39.662374 21863 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 22:50:39.662386 21860 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 22:50:39.662824 21765 server_base.cc:1034] running on GCE node
I20260929 22:50:39.663030 21765 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 22:50:39.663261 21765 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 22:50:39.664405 21765 hybrid_clock.cc:648] HybridClock initialized: now 1790722239664390 us; error 30 us; skew 500 ppm
Sep 29 22:50:39 dist-test-slave-98mk krb5kdc[20968](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790722239, etypes {rep=17 tkt=17 ses=17}, oryx/127.14.147.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 22:50:39.667177 21765 init.cc:377] Logged in from keytab as oryx/127.14.147.193@KRBTEST.COM (short username oryx)
I20260929 22:50:39.667804 21765 webserver.cc:460] Webserver started at http://127.14.147.193:41813/ using document root <none> and password file <none>
I20260929 22:50:39.668042 21765 fs_manager.cc:359] Metadata directory not provided
I20260929 22:50:39.668115 21765 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 22:50:39.669767 21765 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.002s sys 0.000s
I20260929 22:50:39.670424 21871 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 22:50:39.670588 21765 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260929 22:50:39.670683 21765 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790722213368738-14927-0/minicluster-data/ts-0/data,/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790722213368738-14927-0/minicluster-data/ts-0/wal
uuid: "e0d74c61684f40338bc384332f90e9c7"
format_stamp: "Formatted at 2026-09-29 22:50:38 on dist-test-slave-98mk"
server_key: "478a426444f2df994ac591f1b7a8a906"
server_key_iv: "87a860fe59b1ad8c2d4e452a67005836"
server_key_version: "encryptionkey@0"
I20260929 22:50:39.671136 21765 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790722213368738-14927-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790722213368738-14927-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790722213368738-14927-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 22:50:39.688593 21765 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 22:50:39.690934 21765 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 22:50:39.691071 21765 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 22:50:39.691322 21765 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 22:50:39.691960 21765 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 22:50:39.692026 21765 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 22:50:39.692055 21765 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 22:50:39.692080 21765 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 22:50:39.697885 21765 rpc_server.cc:307] RPC server started. Bound to: 127.14.147.193:44967
I20260929 22:50:39.697954 21984 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.147.193:44967 every 8 connection(s)
I20260929 22:50:39.698298 21765 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790722213368738-14927-0/minicluster-data/ts-0/data/info.pb
I20260929 22:50:39.699482 14927 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhWOCLk/build/release/bin/kudu as pid 21765
I20260929 22:50:39.699644 14927 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhWOCLk/build/release/bin/kudu with pid 21212
I20260929 22:50:39.704208 14927 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhWOCLk/build/release/bin/kudu
/tmp/dist-test-taskhWOCLk/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790722213368738-14927-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790722213368738-14927-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790722213368738-14927-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790722213368738-14927-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.147.194:45475
--local_ip_for_outbound_sockets=127.14.147.194
--tserver_master_addrs=127.14.147.254:45601
--webserver_port=42563
--webserver_interface=127.14.147.194
--keytab_file=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790722213368738-14927-0/krb5kdc/oryx_127.14.147.194.keytab
--principal=oryx/127.14.147.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.147.212:34343
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.14.147.254:45601,127.14.147.253:40121
--tserver_master_addrs=127.14.147.254:45601,127.14.147.253:40121,127.14.147.252:41485 with env {KRB5CCNAME=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790722213368738-14927-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790722213368738-14927-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790722213368738-14927-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 29 22:50:39 dist-test-slave-98mk krb5kdc[20968](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790722239, etypes {rep=17 tkt=17 ses=17}, oryx/127.14.147.193@KRBTEST.COM for oryx/127.14.147.254@KRBTEST.COM
Sep 29 22:50:39 dist-test-slave-98mk krb5kdc[20968](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790722239, etypes {rep=17 tkt=17 ses=17}, oryx/127.14.147.193@KRBTEST.COM for oryx/127.14.147.253@KRBTEST.COM
Sep 29 22:50:39 dist-test-slave-98mk krb5kdc[20968](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790722239, etypes {rep=17 tkt=17 ses=17}, oryx/127.14.147.193@KRBTEST.COM for oryx/127.14.147.252@KRBTEST.COM
I20260929 22:50:39.712564 21668 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:50:39.699379 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.147.193:51551 (local address 127.14.147.253:40121)
0929 22:50:39.699457 (+ 78us) server_negotiation.cc:206] Beginning negotiation
0929 22:50:39.699459 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 22:50:39.699751 (+ 292us) server_negotiation.cc:407] Connection header received
0929 22:50:39.702194 (+ 2443us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 22:50:39.702196 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 22:50:39.702209 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 22:50:39.702239 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0929 22:50:39.703404 (+ 1165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:50:39.704192 (+ 788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:50:39.705689 (+ 1497us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:50:39.706001 (+ 312us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:50:39.706039 (+ 38us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 22:50:39.709389 (+ 3350us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 22:50:39.709391 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 22:50:39.709391 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 22:50:39.709399 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0929 22:50:39.710832 (+ 1433us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 22:50:39.711464 (+ 632us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 22:50:39.711465 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 22:50:39.711466 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 22:50:39.711523 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 22:50:39.711815 (+ 292us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 22:50:39.711816 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 22:50:39.711816 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 22:50:39.711966 (+ 150us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 22:50:39.712017 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0929 22:50:39.712389 (+ 372us) server_negotiation.cc:299] Negotiation successful
0929 22:50:39.712469 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260929 22:50:39.713008 21073 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:50:39.699258 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.147.193:47209 (local address 127.14.147.254:45601)
0929 22:50:39.699309 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0929 22:50:39.699311 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 22:50:39.699485 (+ 174us) server_negotiation.cc:407] Connection header received
0929 22:50:39.702167 (+ 2682us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 22:50:39.702168 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 22:50:39.702182 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 22:50:39.702216 (+ 34us) server_negotiation.cc:226] Negotiated authn=SASL
0929 22:50:39.703494 (+ 1278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:50:39.704165 (+ 671us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:50:39.705732 (+ 1567us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:50:39.706029 (+ 297us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:50:39.706070 (+ 41us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 22:50:39.710365 (+ 4295us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 22:50:39.710367 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 22:50:39.710367 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 22:50:39.710376 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0929 22:50:39.711894 (+ 1518us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 22:50:39.712262 (+ 368us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 22:50:39.712263 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 22:50:39.712263 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 22:50:39.712320 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 22:50:39.712503 (+ 183us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 22:50:39.712504 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 22:50:39.712504 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 22:50:39.712625 (+ 121us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 22:50:39.712671 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0929 22:50:39.712848 (+ 177us) server_negotiation.cc:299] Negotiation successful
0929 22:50:39.712926 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260929 22:50:39.713610 21991 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:50:39.699058 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.147.253:40121 (local address 127.14.147.193:51551)
0929 22:50:39.699680 (+ 622us) negotiation.cc:107] Waiting for socket to connect
0929 22:50:39.699689 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0929 22:50:39.702048 (+ 2359us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 22:50:39.702415 (+ 367us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 22:50:39.702420 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 22:50:39.703038 (+ 618us) client_negotiation.cc:189] Negotiated authn=SASL
0929 22:50:39.703278 (+ 240us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 22:50:39.703285 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 22:50:39.704292 (+ 1007us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 22:50:39.704293 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 22:50:39.705595 (+ 1302us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 22:50:39.705597 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 22:50:39.706129 (+ 532us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 22:50:39.706129 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 22:50:39.706266 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 22:50:39.707007 (+ 741us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 22:50:39.707011 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0929 22:50:39.709294 (+ 2283us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 22:50:39.711083 (+ 1789us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 22:50:39.711089 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 22:50:39.711091 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 22:50:39.711347 (+ 256us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 22:50:39.711593 (+ 246us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 22:50:39.711594 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 22:50:39.711594 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 22:50:39.711736 (+ 142us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 22:50:39.712047 (+ 311us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 22:50:39.712053 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 22:50:39.712221 (+ 168us) client_negotiation.cc:769] Sending connection context
0929 22:50:39.712935 (+ 714us) client_negotiation.cc:240] Negotiation successful
0929 22:50:39.713356 (+ 421us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":596,"mutex_wait_us":52,"thread_start_us":85,"threads_started":1}
I20260929 22:50:39.713610 21989 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:50:39.699058 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.147.254:45601 (local address 127.14.147.193:47209)
0929 22:50:39.699330 (+ 272us) negotiation.cc:107] Waiting for socket to connect
0929 22:50:39.699375 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0929 22:50:39.702039 (+ 2664us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 22:50:39.702415 (+ 376us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 22:50:39.702420 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 22:50:39.703359 (+ 939us) client_negotiation.cc:189] Negotiated authn=SASL
0929 22:50:39.703412 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 22:50:39.703432 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 22:50:39.704555 (+ 1123us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 22:50:39.704556 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 22:50:39.705637 (+ 1081us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 22:50:39.705640 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 22:50:39.706208 (+ 568us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 22:50:39.706209 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 22:50:39.706273 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 22:50:39.706880 (+ 607us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 22:50:39.706891 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0929 22:50:39.710219 (+ 3328us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 22:50:39.711967 (+ 1748us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 22:50:39.711968 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 22:50:39.711969 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 22:50:39.712168 (+ 199us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 22:50:39.712386 (+ 218us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 22:50:39.712387 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 22:50:39.712387 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 22:50:39.712439 (+ 52us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 22:50:39.712703 (+ 264us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 22:50:39.712704 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 22:50:39.712747 (+ 43us) client_negotiation.cc:769] Sending connection context
0929 22:50:39.713130 (+ 383us) client_negotiation.cc:240] Negotiation successful
0929 22:50:39.713434 (+ 304us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":229,"mutex_wait_us":199,"thread_start_us":95,"threads_started":1}
I20260929 22:50:39.714264 21986 heartbeater.cc:347] Connected to a master server at 127.14.147.254:45601
I20260929 22:50:39.714366 21986 heartbeater.cc:464] Registering TS with master...
I20260929 22:50:39.714411 21987 heartbeater.cc:347] Connected to a master server at 127.14.147.253:40121
I20260929 22:50:39.714439 21987 heartbeater.cc:464] Registering TS with master...
I20260929 22:50:39.714457 21992 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:50:39.699139 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.147.193:47325 (local address 127.14.147.252:41485)
0929 22:50:39.699548 (+ 409us) server_negotiation.cc:206] Beginning negotiation
0929 22:50:39.699550 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 22:50:39.699560 (+ 10us) server_negotiation.cc:407] Connection header received
0929 22:50:39.702166 (+ 2606us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 22:50:39.702168 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 22:50:39.702182 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 22:50:39.702229 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0929 22:50:39.703523 (+ 1294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:50:39.704532 (+ 1009us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:50:39.705688 (+ 1156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:50:39.706001 (+ 313us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:50:39.706039 (+ 38us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 22:50:39.711003 (+ 4964us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 22:50:39.711005 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 22:50:39.711005 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 22:50:39.711014 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0929 22:50:39.712915 (+ 1901us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 22:50:39.713420 (+ 505us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 22:50:39.713422 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 22:50:39.713422 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 22:50:39.713489 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 22:50:39.713716 (+ 227us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 22:50:39.713717 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 22:50:39.713718 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 22:50:39.713859 (+ 141us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 22:50:39.713911 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0929 22:50:39.714267 (+ 356us) server_negotiation.cc:299] Negotiation successful
0929 22:50:39.714360 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":387,"thread_start_us":67,"threads_started":1}
I20260929 22:50:39.714560 21986 heartbeater.cc:511] Master 127.14.147.254:45601 requested a full tablet report, sending...
I20260929 22:50:39.714560 21987 heartbeater.cc:511] Master 127.14.147.253:40121 requested a full tablet report, sending...
I20260929 22:50:39.714999 21014 ts_manager.cc:194] Re-registered known tserver with Master: e0d74c61684f40338bc384332f90e9c7 (127.14.147.193:44967)
I20260929 22:50:39.715287 21617 ts_manager.cc:194] Registered new tserver with Master: e0d74c61684f40338bc384332f90e9c7 (127.14.147.193:44967)
I20260929 22:50:39.715516 21990 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:50:39.699058 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.147.252:41485 (local address 127.14.147.193:47325)
0929 22:50:39.699486 (+ 428us) negotiation.cc:107] Waiting for socket to connect
0929 22:50:39.699495 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0929 22:50:39.702030 (+ 2535us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 22:50:39.702415 (+ 385us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 22:50:39.702420 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 22:50:39.702975 (+ 555us) client_negotiation.cc:189] Negotiated authn=SASL
0929 22:50:39.703278 (+ 303us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 22:50:39.703285 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 22:50:39.704612 (+ 1327us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 22:50:39.704613 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 22:50:39.705595 (+ 982us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 22:50:39.705597 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 22:50:39.706166 (+ 569us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 22:50:39.706167 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 22:50:39.706265 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 22:50:39.706918 (+ 653us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 22:50:39.706923 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0929 22:50:39.710226 (+ 3303us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 22:50:39.713011 (+ 2785us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 22:50:39.713012 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 22:50:39.713013 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 22:50:39.713325 (+ 312us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 22:50:39.713558 (+ 233us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 22:50:39.713559 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 22:50:39.713559 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 22:50:39.713639 (+ 80us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 22:50:39.713946 (+ 307us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 22:50:39.713948 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 22:50:39.713990 (+ 42us) client_negotiation.cc:769] Sending connection context
0929 22:50:39.714046 (+ 56us) client_negotiation.cc:240] Negotiation successful
0929 22:50:39.714151 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":400,"mutex_wait_us":77,"thread_start_us":81,"threads_started":1}
I20260929 22:50:39.715567 21014 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.14.147.193@KRBTEST.COM'} at 127.14.147.193:47209
I20260929 22:50:39.715858 21985 heartbeater.cc:347] Connected to a master server at 127.14.147.252:41485
I20260929 22:50:39.715910 21985 heartbeater.cc:464] Registering TS with master...
I20260929 22:50:39.716005 21985 heartbeater.cc:511] Master 127.14.147.252:41485 requested a full tablet report, sending...
I20260929 22:50:39.716320 21805 ts_manager.cc:194] Registered new tserver with Master: e0d74c61684f40338bc384332f90e9c7 (127.14.147.193:44967)
W20260929 22:50:39.787732 21993 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 22:50:39.787951 21993 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 22:50:39.787981 21993 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 22:50:39.789405 21993 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 22:50:39.789464 21993 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 22:50:39.789492 21993 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.147.194
I20260929 22:50:39.791150 21993 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.147.212:34343
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790722213368738-14927-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790722213368738-14927-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790722213368738-14927-0/krb5kdc/oryx_127.14.147.194.keytab
--principal=oryx/127.14.147.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.147.194:45475
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790722213368738-14927-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.14.147.194
--webserver_port=42563
--tserver_master_addrs=127.14.147.254:45601,127.14.147.253:40121,127.14.147.252:41485
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21993
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.147.194
--log_dir=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790722213368738-14927-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 21:58:54 UTC on ccycloud.quasar-xguoti.root.comops.site
build id 4681390
I20260929 22:50:39.791404 21993 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 22:50:39.791683 21993 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 22:50:39.796619 21999 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 22:50:39.796742 21993 server_base.cc:1034] running on GCE node
W20260929 22:50:39.796619 22001 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 22:50:39.796833 21998 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 22:50:39.797032 21993 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 22:50:39.797271 21993 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 22:50:39.798454 21993 hybrid_clock.cc:648] HybridClock initialized: now 1790722239798422 us; error 46 us; skew 500 ppm
Sep 29 22:50:39 dist-test-slave-98mk krb5kdc[20968](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790722239, etypes {rep=17 tkt=17 ses=17}, oryx/127.14.147.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 22:50:39.800889 21993 init.cc:377] Logged in from keytab as oryx/127.14.147.194@KRBTEST.COM (short username oryx)
I20260929 22:50:39.801328 21993 webserver.cc:460] Webserver started at http://127.14.147.194:42563/ using document root <none> and password file <none>
I20260929 22:50:39.801519 21993 fs_manager.cc:359] Metadata directory not provided
I20260929 22:50:39.801571 21993 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 22:50:39.802817 21993 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260929 22:50:39.803367 22008 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 22:50:39.803539 21993 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260929 22:50:39.803614 21993 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790722213368738-14927-0/minicluster-data/ts-1/data,/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790722213368738-14927-0/minicluster-data/ts-1/wal
uuid: "ab4a15db2c3f422e8646d4b5255cf8df"
format_stamp: "Formatted at 2026-09-29 22:50:38 on dist-test-slave-98mk"
server_key: "1efd4d2d11fa0ea992729dde3a4bc74b"
server_key_iv: "f8771db19f7a73734110f71bc7974c91"
server_key_version: "encryptionkey@0"
I20260929 22:50:39.803972 21993 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790722213368738-14927-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790722213368738-14927-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790722213368738-14927-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 22:50:39.812083 21993 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 22:50:39.814153 21993 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 22:50:39.814268 21993 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 22:50:39.814464 21993 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 22:50:39.814734 21993 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 22:50:39.814769 21993 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 22:50:39.814790 21993 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 22:50:39.814806 21993 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 22:50:39.820691 21993 rpc_server.cc:307] RPC server started. Bound to: 127.14.147.194:45475
I20260929 22:50:39.820827 22121 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.147.194:45475 every 8 connection(s)
I20260929 22:50:39.821115 21993 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790722213368738-14927-0/minicluster-data/ts-1/data/info.pb
Sep 29 22:50:39 dist-test-slave-98mk krb5kdc[20968](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790722239, etypes {rep=17 tkt=17 ses=17}, oryx/127.14.147.194@KRBTEST.COM for oryx/127.14.147.253@KRBTEST.COM
Sep 29 22:50:39 dist-test-slave-98mk krb5kdc[20968](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790722239, etypes {rep=17 tkt=17 ses=17}, oryx/127.14.147.194@KRBTEST.COM for oryx/127.14.147.252@KRBTEST.COM
Sep 29 22:50:39 dist-test-slave-98mk krb5kdc[20968](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790722239, etypes {rep=17 tkt=17 ses=17}, oryx/127.14.147.194@KRBTEST.COM for oryx/127.14.147.254@KRBTEST.COM
I20260929 22:50:39.829773 14927 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhWOCLk/build/release/bin/kudu as pid 21993
I20260929 22:50:39.829942 14927 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhWOCLk/build/release/bin/kudu with pid 21347
I20260929 22:50:39.832913 21073 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:50:39.821895 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.147.194:43673 (local address 127.14.147.254:45601)
0929 22:50:39.821947 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0929 22:50:39.821950 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 22:50:39.822301 (+ 351us) server_negotiation.cc:407] Connection header received
0929 22:50:39.823172 (+ 871us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 22:50:39.823173 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 22:50:39.823186 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 22:50:39.823236 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0929 22:50:39.824057 (+ 821us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:50:39.824849 (+ 792us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:50:39.826829 (+ 1980us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:50:39.827086 (+ 257us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:50:39.827153 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 22:50:39.829685 (+ 2532us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 22:50:39.829687 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 22:50:39.829687 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 22:50:39.829694 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0929 22:50:39.831449 (+ 1755us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 22:50:39.831824 (+ 375us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 22:50:39.831825 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 22:50:39.831825 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 22:50:39.831886 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 22:50:39.832166 (+ 280us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 22:50:39.832167 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 22:50:39.832168 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 22:50:39.832332 (+ 164us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 22:50:39.832391 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0929 22:50:39.832722 (+ 331us) server_negotiation.cc:299] Negotiation successful
0929 22:50:39.832813 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260929 22:50:39.833148 22126 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:50:39.821836 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.147.252:41485 (local address 127.14.147.194:36589)
0929 22:50:39.822048 (+ 212us) negotiation.cc:107] Waiting for socket to connect
0929 22:50:39.822090 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0929 22:50:39.823057 (+ 967us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 22:50:39.823310 (+ 253us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 22:50:39.823315 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 22:50:39.823749 (+ 434us) client_negotiation.cc:189] Negotiated authn=SASL
0929 22:50:39.823972 (+ 223us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 22:50:39.823978 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 22:50:39.824916 (+ 938us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 22:50:39.824917 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 22:50:39.825979 (+ 1062us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 22:50:39.825982 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 22:50:39.826713 (+ 731us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 22:50:39.826714 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 22:50:39.826799 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 22:50:39.827350 (+ 551us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 22:50:39.827354 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0929 22:50:39.829579 (+ 2225us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 22:50:39.831885 (+ 2306us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 22:50:39.831886 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 22:50:39.831886 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 22:50:39.832060 (+ 174us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 22:50:39.832251 (+ 191us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 22:50:39.832252 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 22:50:39.832252 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 22:50:39.832305 (+ 53us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 22:50:39.832550 (+ 245us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 22:50:39.832551 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 22:50:39.832583 (+ 32us) client_negotiation.cc:769] Sending connection context
0929 22:50:39.832684 (+ 101us) client_negotiation.cc:240] Negotiation successful
0929 22:50:39.832886 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":172,"mutex_wait_us":44,"thread_start_us":79,"threads_started":1}
I20260929 22:50:39.833148 22127 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:50:39.821836 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.147.254:45601 (local address 127.14.147.194:43673)
0929 22:50:39.822237 (+ 401us) negotiation.cc:107] Waiting for socket to connect
0929 22:50:39.822246 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0929 22:50:39.823064 (+ 818us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 22:50:39.823309 (+ 245us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 22:50:39.823313 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 22:50:39.823748 (+ 435us) client_negotiation.cc:189] Negotiated authn=SASL
0929 22:50:39.823970 (+ 222us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 22:50:39.823975 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 22:50:39.826079 (+ 2104us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 22:50:39.826079 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 22:50:39.826710 (+ 631us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 22:50:39.826713 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 22:50:39.827153 (+ 440us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 22:50:39.827154 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 22:50:39.827210 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 22:50:39.827559 (+ 349us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 22:50:39.827564 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0929 22:50:39.829579 (+ 2015us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 22:50:39.831516 (+ 1937us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 22:50:39.831525 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 22:50:39.831528 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 22:50:39.831756 (+ 228us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 22:50:39.831955 (+ 199us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 22:50:39.831956 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 22:50:39.831956 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 22:50:39.832086 (+ 130us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 22:50:39.832415 (+ 329us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 22:50:39.832422 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 22:50:39.832564 (+ 142us) client_negotiation.cc:769] Sending connection context
0929 22:50:39.832684 (+ 120us) client_negotiation.cc:240] Negotiation successful
0929 22:50:39.832950 (+ 266us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":371,"mutex_wait_us":372,"thread_start_us":67,"threads_started":1}
I20260929 22:50:39.833678 21992 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:50:39.822074 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.147.194:36589 (local address 127.14.147.252:41485)
0929 22:50:39.822134 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0929 22:50:39.822136 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 22:50:39.822149 (+ 13us) server_negotiation.cc:407] Connection header received
0929 22:50:39.823179 (+ 1030us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 22:50:39.823180 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 22:50:39.823195 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 22:50:39.823223 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0929 22:50:39.824086 (+ 863us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:50:39.824837 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:50:39.826421 (+ 1584us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:50:39.826618 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:50:39.826675 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 22:50:39.829698 (+ 3023us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 22:50:39.829699 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 22:50:39.829699 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 22:50:39.829709 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0929 22:50:39.831809 (+ 2100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 22:50:39.832126 (+ 317us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 22:50:39.832127 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 22:50:39.832127 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 22:50:39.832188 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 22:50:39.832365 (+ 177us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 22:50:39.832366 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 22:50:39.832366 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 22:50:39.832491 (+ 125us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 22:50:39.832552 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0929 22:50:39.833488 (+ 936us) server_negotiation.cc:299] Negotiation successful
0929 22:50:39.833588 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260929 22:50:39.833668 22123 heartbeater.cc:347] Connected to a master server at 127.14.147.254:45601
I20260929 22:50:39.833758 22123 heartbeater.cc:464] Registering TS with master...
I20260929 22:50:39.833940 22123 heartbeater.cc:511] Master 127.14.147.254:45601 requested a full tablet report, sending...
I20260929 22:50:39.834391 21014 ts_manager.cc:194] Re-registered known tserver with Master: ab4a15db2c3f422e8646d4b5255cf8df (127.14.147.194:45475)
I20260929 22:50:39.834851 21014 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.14.147.194@KRBTEST.COM'} at 127.14.147.194:43673
I20260929 22:50:39.836069 22122 heartbeater.cc:347] Connected to a master server at 127.14.147.252:41485
I20260929 22:50:39.836130 22122 heartbeater.cc:464] Registering TS with master...
I20260929 22:50:39.836202 22122 heartbeater.cc:511] Master 127.14.147.252:41485 requested a full tablet report, sending...
I20260929 22:50:39.836514 21805 ts_manager.cc:194] Registered new tserver with Master: ab4a15db2c3f422e8646d4b5255cf8df (127.14.147.194:45475)
I20260929 22:50:39.837316 14927 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhWOCLk/build/release/bin/kudu
/tmp/dist-test-taskhWOCLk/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790722213368738-14927-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790722213368738-14927-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790722213368738-14927-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790722213368738-14927-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.147.195:38905
--local_ip_for_outbound_sockets=127.14.147.195
--tserver_master_addrs=127.14.147.254:45601
--webserver_port=45097
--webserver_interface=127.14.147.195
--keytab_file=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790722213368738-14927-0/krb5kdc/oryx_127.14.147.195.keytab
--principal=oryx/127.14.147.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.147.212:34343
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.14.147.254:45601,127.14.147.253:40121
--tserver_master_addrs=127.14.147.254:45601,127.14.147.253:40121,127.14.147.252:41485 with env {KRB5CCNAME=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790722213368738-14927-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790722213368738-14927-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790722213368738-14927-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 22:50:39.837779 21668 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:50:39.821901 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.147.194:42155 (local address 127.14.147.253:40121)
0929 22:50:39.821948 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0929 22:50:39.821950 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 22:50:39.822288 (+ 338us) server_negotiation.cc:407] Connection header received
0929 22:50:39.823180 (+ 892us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 22:50:39.823181 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 22:50:39.823193 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 22:50:39.823224 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0929 22:50:39.824121 (+ 897us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:50:39.824615 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:50:39.826093 (+ 1478us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:50:39.826307 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:50:39.826368 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 22:50:39.828829 (+ 2461us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 22:50:39.828830 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 22:50:39.828830 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 22:50:39.828837 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0929 22:50:39.836677 (+ 7840us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 22:50:39.837020 (+ 343us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 22:50:39.837021 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 22:50:39.837022 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 22:50:39.837090 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 22:50:39.837267 (+ 177us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 22:50:39.837268 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 22:50:39.837268 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 22:50:39.837400 (+ 132us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 22:50:39.837464 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0929 22:50:39.837609 (+ 145us) server_negotiation.cc:299] Negotiation successful
0929 22:50:39.837690 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260929 22:50:39.837831 22128 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:50:39.821836 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.147.253:40121 (local address 127.14.147.194:42155)
0929 22:50:39.822220 (+ 384us) negotiation.cc:107] Waiting for socket to connect
0929 22:50:39.822229 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0929 22:50:39.823063 (+ 834us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 22:50:39.823310 (+ 247us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 22:50:39.823315 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 22:50:39.823758 (+ 443us) client_negotiation.cc:189] Negotiated authn=SASL
0929 22:50:39.823972 (+ 214us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 22:50:39.823978 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 22:50:39.824720 (+ 742us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 22:50:39.824721 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 22:50:39.825979 (+ 1258us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 22:50:39.825982 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 22:50:39.826392 (+ 410us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 22:50:39.826393 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 22:50:39.826530 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 22:50:39.827077 (+ 547us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 22:50:39.827088 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0929 22:50:39.828739 (+ 1651us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 22:50:39.836742 (+ 8003us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 22:50:39.836742 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 22:50:39.836743 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 22:50:39.836951 (+ 208us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 22:50:39.837151 (+ 200us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 22:50:39.837152 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 22:50:39.837152 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 22:50:39.837214 (+ 62us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 22:50:39.837470 (+ 256us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 22:50:39.837471 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 22:50:39.837516 (+ 45us) client_negotiation.cc:769] Sending connection context
0929 22:50:39.837576 (+ 60us) client_negotiation.cc:240] Negotiation successful
0929 22:50:39.837731 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":356,"mutex_wait_us":12,"thread_start_us":90,"threads_started":1}
I20260929 22:50:39.838137 22124 heartbeater.cc:347] Connected to a master server at 127.14.147.253:40121
I20260929 22:50:39.838186 22124 heartbeater.cc:464] Registering TS with master...
I20260929 22:50:39.838225 22124 heartbeater.cc:511] Master 127.14.147.253:40121 requested a full tablet report, sending...
I20260929 22:50:39.838492 21617 ts_manager.cc:194] Registered new tserver with Master: ab4a15db2c3f422e8646d4b5255cf8df (127.14.147.194:45475)
W20260929 22:50:39.918988 22130 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 22:50:39.919217 22130 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 22:50:39.919242 22130 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 22:50:39.920523 22130 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 22:50:39.920579 22130 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 22:50:39.920608 22130 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.147.195
I20260929 22:50:39.922161 22130 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.147.212:34343
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790722213368738-14927-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790722213368738-14927-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790722213368738-14927-0/krb5kdc/oryx_127.14.147.195.keytab
--principal=oryx/127.14.147.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.147.195:38905
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790722213368738-14927-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.14.147.195
--webserver_port=45097
--tserver_master_addrs=127.14.147.254:45601,127.14.147.253:40121,127.14.147.252:41485
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22130
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.147.195
--log_dir=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790722213368738-14927-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 21:58:54 UTC on ccycloud.quasar-xguoti.root.comops.site
build id 4681390
I20260929 22:50:39.922411 22130 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 22:50:39.922650 22130 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 22:50:39.927632 22136 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 22:50:39.927616 22138 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 22:50:39.927611 22135 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 22:50:39.927651 22130 server_base.cc:1034] running on GCE node
I20260929 22:50:39.927996 22130 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 22:50:39.928210 22130 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 22:50:39.929358 22130 hybrid_clock.cc:648] HybridClock initialized: now 1790722239929344 us; error 36 us; skew 500 ppm
Sep 29 22:50:39 dist-test-slave-98mk krb5kdc[20968](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790722239, etypes {rep=17 tkt=17 ses=17}, oryx/127.14.147.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 22:50:39.932158 22130 init.cc:377] Logged in from keytab as oryx/127.14.147.195@KRBTEST.COM (short username oryx)
I20260929 22:50:39.932593 22130 webserver.cc:460] Webserver started at http://127.14.147.195:45097/ using document root <none> and password file <none>
I20260929 22:50:39.932823 22130 fs_manager.cc:359] Metadata directory not provided
I20260929 22:50:39.932885 22130 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 22:50:39.934279 22130 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260929 22:50:39.934993 22145 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 22:50:39.935177 22130 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260929 22:50:39.935269 22130 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790722213368738-14927-0/minicluster-data/ts-2/data,/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790722213368738-14927-0/minicluster-data/ts-2/wal
uuid: "f9ed52b3ef574d3d8c17014dbf842a54"
format_stamp: "Formatted at 2026-09-29 22:50:39 on dist-test-slave-98mk"
server_key: "b96fa853a9eaa6a848b9328200c3a5fd"
server_key_iv: "5bfb613a395ad47c0a7386f0eadced5a"
server_key_version: "encryptionkey@0"
I20260929 22:50:39.935767 22130 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790722213368738-14927-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790722213368738-14927-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790722213368738-14927-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 22:50:39.961602 22130 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 22:50:39.963388 22130 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 22:50:39.963559 22130 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 22:50:39.963804 22130 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 22:50:39.964108 22130 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 22:50:39.964139 22130 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 22:50:39.964162 22130 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 22:50:39.964186 22130 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 22:50:39.969736 22130 rpc_server.cc:307] RPC server started. Bound to: 127.14.147.195:38905
I20260929 22:50:39.969810 22258 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.147.195:38905 every 8 connection(s)
I20260929 22:50:39.970101 22130 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790722213368738-14927-0/minicluster-data/ts-2/data/info.pb
I20260929 22:50:39.972625 14927 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhWOCLk/build/release/bin/kudu as pid 22130
Sep 29 22:50:39 dist-test-slave-98mk krb5kdc[20968](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790722239, etypes {rep=17 tkt=17 ses=17}, oryx/127.14.147.195@KRBTEST.COM for oryx/127.14.147.252@KRBTEST.COM
Sep 29 22:50:39 dist-test-slave-98mk krb5kdc[20968](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790722239, etypes {rep=17 tkt=17 ses=17}, oryx/127.14.147.195@KRBTEST.COM for oryx/127.14.147.253@KRBTEST.COM
Sep 29 22:50:39 dist-test-slave-98mk krb5kdc[20968](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790722239, etypes {rep=17 tkt=17 ses=17}, oryx/127.14.147.195@KRBTEST.COM for oryx/127.14.147.254@KRBTEST.COM
I20260929 22:50:39.983892 21992 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:50:39.970969 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.147.195:54581 (local address 127.14.147.252:41485)
0929 22:50:39.971010 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0929 22:50:39.971011 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0929 22:50:39.971392 (+ 381us) server_negotiation.cc:407] Connection header received
0929 22:50:39.972427 (+ 1035us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 22:50:39.972429 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 22:50:39.972441 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 22:50:39.972499 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0929 22:50:39.973462 (+ 963us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:50:39.974241 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:50:39.975773 (+ 1532us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:50:39.976041 (+ 268us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:50:39.976096 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 22:50:39.980827 (+ 4731us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 22:50:39.980828 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 22:50:39.980829 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 22:50:39.980835 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0929 22:50:39.982335 (+ 1500us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 22:50:39.982790 (+ 455us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 22:50:39.982792 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 22:50:39.982792 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 22:50:39.982843 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 22:50:39.983102 (+ 259us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 22:50:39.983102 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 22:50:39.983103 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 22:50:39.983239 (+ 136us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 22:50:39.983290 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0929 22:50:39.983692 (+ 402us) server_negotiation.cc:299] Negotiation successful
0929 22:50:39.983783 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260929 22:50:39.984033 22265 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:50:39.970809 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.147.252:41485 (local address 127.14.147.195:54581)
0929 22:50:39.971333 (+ 524us) negotiation.cc:107] Waiting for socket to connect
0929 22:50:39.971341 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0929 22:50:39.972332 (+ 991us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 22:50:39.972584 (+ 252us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 22:50:39.972589 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 22:50:39.973068 (+ 479us) client_negotiation.cc:189] Negotiated authn=SASL
0929 22:50:39.973355 (+ 287us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 22:50:39.973362 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 22:50:39.974394 (+ 1032us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 22:50:39.974395 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 22:50:39.975701 (+ 1306us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 22:50:39.975703 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 22:50:39.976172 (+ 469us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 22:50:39.976173 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 22:50:39.976310 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 22:50:39.978792 (+ 2482us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 22:50:39.978803 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0929 22:50:39.980721 (+ 1918us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 22:50:39.982408 (+ 1687us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 22:50:39.982416 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 22:50:39.982418 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 22:50:39.982697 (+ 279us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 22:50:39.982907 (+ 210us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 22:50:39.982908 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 22:50:39.982908 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 22:50:39.983034 (+ 126us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 22:50:39.983326 (+ 292us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 22:50:39.983333 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 22:50:39.983521 (+ 188us) client_negotiation.cc:769] Sending connection context
0929 22:50:39.983626 (+ 105us) client_negotiation.cc:240] Negotiation successful
0929 22:50:39.983798 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":502,"mutex_wait_us":153,"thread_start_us":86,"threads_started":1}
I20260929 22:50:39.984272 22272 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:50:39.974159 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60050 (local address 127.14.147.252:41485)
0929 22:50:39.974839 (+ 680us) server_negotiation.cc:206] Beginning negotiation
0929 22:50:39.974841 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 22:50:39.975156 (+ 315us) server_negotiation.cc:407] Connection header received
0929 22:50:39.975194 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 22:50:39.975195 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 22:50:39.975206 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 22:50:39.975262 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0929 22:50:39.975957 (+ 695us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:50:39.976599 (+ 642us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:50:39.977672 (+ 1073us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:50:39.977963 (+ 291us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:50:39.978023 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 22:50:39.980985 (+ 2962us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 22:50:39.980986 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 22:50:39.980987 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 22:50:39.980992 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0929 22:50:39.982993 (+ 2001us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 22:50:39.983333 (+ 340us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 22:50:39.983333 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 22:50:39.983334 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 22:50:39.983384 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 22:50:39.983585 (+ 201us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 22:50:39.983585 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 22:50:39.983586 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 22:50:39.983698 (+ 112us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 22:50:39.983798 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0929 22:50:39.984114 (+ 316us) server_negotiation.cc:299] Negotiation successful
0929 22:50:39.984192 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":661,"thread_start_us":50,"threads_started":1}
I20260929 22:50:39.984548 22259 heartbeater.cc:347] Connected to a master server at 127.14.147.252:41485
I20260929 22:50:39.984658 22259 heartbeater.cc:464] Registering TS with master...
I20260929 22:50:39.984860 22259 heartbeater.cc:511] Master 127.14.147.252:41485 requested a full tablet report, sending...
I20260929 22:50:39.985284 21805 ts_manager.cc:194] Registered new tserver with Master: f9ed52b3ef574d3d8c17014dbf842a54 (127.14.147.195:38905)
I20260929 22:50:39.986244 21073 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:50:39.971118 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.147.195:48659 (local address 127.14.147.254:45601)
0929 22:50:39.971176 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0929 22:50:39.971177 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0929 22:50:39.971206 (+ 29us) server_negotiation.cc:407] Connection header received
0929 22:50:39.972684 (+ 1478us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 22:50:39.972685 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 22:50:39.972696 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 22:50:39.972742 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0929 22:50:39.973462 (+ 720us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:50:39.974246 (+ 784us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:50:39.977879 (+ 3633us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:50:39.978176 (+ 297us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:50:39.978214 (+ 38us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 22:50:39.981984 (+ 3770us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 22:50:39.981985 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 22:50:39.981985 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 22:50:39.981992 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0929 22:50:39.985177 (+ 3185us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 22:50:39.985527 (+ 350us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 22:50:39.985528 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 22:50:39.985528 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 22:50:39.985593 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 22:50:39.985780 (+ 187us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 22:50:39.985780 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 22:50:39.985781 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 22:50:39.985912 (+ 131us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 22:50:39.985971 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0929 22:50:39.986084 (+ 113us) server_negotiation.cc:299] Negotiation successful
0929 22:50:39.986157 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260929 22:50:39.986272 22264 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:50:39.970809 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.147.254:45601 (local address 127.14.147.195:48659)
0929 22:50:39.971114 (+ 305us) negotiation.cc:107] Waiting for socket to connect
0929 22:50:39.971135 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0929 22:50:39.972326 (+ 1191us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 22:50:39.972784 (+ 458us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 22:50:39.972785 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 22:50:39.973108 (+ 323us) client_negotiation.cc:189] Negotiated authn=SASL
0929 22:50:39.973378 (+ 270us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 22:50:39.973379 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 22:50:39.974367 (+ 988us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 22:50:39.974368 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 22:50:39.977781 (+ 3413us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 22:50:39.977784 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 22:50:39.978288 (+ 504us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 22:50:39.978288 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 22:50:39.978371 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 22:50:39.979717 (+ 1346us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 22:50:39.979722 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0929 22:50:39.981910 (+ 2188us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 22:50:39.985238 (+ 3328us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 22:50:39.985239 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 22:50:39.985240 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 22:50:39.985433 (+ 193us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 22:50:39.985656 (+ 223us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 22:50:39.985657 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 22:50:39.985657 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 22:50:39.985713 (+ 56us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 22:50:39.985959 (+ 246us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 22:50:39.985960 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 22:50:39.985998 (+ 38us) client_negotiation.cc:769] Sending connection context
0929 22:50:39.986051 (+ 53us) client_negotiation.cc:240] Negotiation successful
0929 22:50:39.986153 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":282,"mutex_wait_us":2497,"thread_start_us":93,"threads_started":1}
I20260929 22:50:39.986640 22263 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:50:39.970808 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.147.253:40121 (local address 127.14.147.195:45229)
0929 22:50:39.971091 (+ 283us) negotiation.cc:107] Waiting for socket to connect
0929 22:50:39.971135 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0929 22:50:39.972333 (+ 1198us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 22:50:39.972643 (+ 310us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 22:50:39.972644 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 22:50:39.973046 (+ 402us) client_negotiation.cc:189] Negotiated authn=SASL
0929 22:50:39.973356 (+ 310us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 22:50:39.973363 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 22:50:39.974312 (+ 949us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 22:50:39.974313 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 22:50:39.976417 (+ 2104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 22:50:39.976420 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 22:50:39.976872 (+ 452us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 22:50:39.976873 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 22:50:39.976931 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 22:50:39.978906 (+ 1975us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 22:50:39.978911 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0929 22:50:39.981616 (+ 2705us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 22:50:39.985448 (+ 3832us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 22:50:39.985449 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 22:50:39.985449 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 22:50:39.985611 (+ 162us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 22:50:39.985809 (+ 198us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 22:50:39.985810 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 22:50:39.985811 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 22:50:39.985864 (+ 53us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 22:50:39.986169 (+ 305us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 22:50:39.986170 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 22:50:39.986212 (+ 42us) client_negotiation.cc:769] Sending connection context
0929 22:50:39.986264 (+ 52us) client_negotiation.cc:240] Negotiation successful
0929 22:50:39.986361 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":243,"mutex_wait_us":1180,"thread_start_us":97,"threads_started":1}
I20260929 22:50:39.986814 22260 heartbeater.cc:347] Connected to a master server at 127.14.147.254:45601
I20260929 22:50:39.986865 22260 heartbeater.cc:464] Registering TS with master...
I20260929 22:50:39.986938 22260 heartbeater.cc:511] Master 127.14.147.254:45601 requested a full tablet report, sending...
I20260929 22:50:39.986990 21668 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:50:39.970804 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.147.195:45229 (local address 127.14.147.253:40121)
0929 22:50:39.970921 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0929 22:50:39.970923 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 22:50:39.971220 (+ 297us) server_negotiation.cc:407] Connection header received
0929 22:50:39.972557 (+ 1337us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 22:50:39.972558 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 22:50:39.972569 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 22:50:39.972619 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0929 22:50:39.973464 (+ 845us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:50:39.974152 (+ 688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:50:39.976497 (+ 2345us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:50:39.976775 (+ 278us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:50:39.976825 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 22:50:39.981702 (+ 4877us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 22:50:39.981703 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 22:50:39.981703 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 22:50:39.981711 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0929 22:50:39.985374 (+ 3663us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 22:50:39.985685 (+ 311us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 22:50:39.985687 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 22:50:39.985687 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 22:50:39.985741 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 22:50:39.985943 (+ 202us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 22:50:39.985944 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 22:50:39.985944 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 22:50:39.986078 (+ 134us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 22:50:39.986141 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0929 22:50:39.986829 (+ 688us) server_negotiation.cc:299] Negotiation successful
0929 22:50:39.986904 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260929 22:50:39.987154 21014 ts_manager.cc:194] Re-registered known tserver with Master: f9ed52b3ef574d3d8c17014dbf842a54 (127.14.147.195:38905)
I20260929 22:50:39.987623 22261 heartbeater.cc:347] Connected to a master server at 127.14.147.253:40121
I20260929 22:50:39.987668 22261 heartbeater.cc:464] Registering TS with master...
I20260929 22:50:39.987680 21014 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.14.147.195@KRBTEST.COM'} at 127.14.147.195:48659
I20260929 22:50:39.987742 22261 heartbeater.cc:511] Master 127.14.147.253:40121 requested a full tablet report, sending...
I20260929 22:50:39.987994 21617 ts_manager.cc:194] Registered new tserver with Master: f9ed52b3ef574d3d8c17014dbf842a54 (127.14.147.195:38905)
I20260929 22:50:39.988436 22274 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:50:39.974644 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56814 (local address 127.14.147.253:40121)
0929 22:50:39.974747 (+ 103us) server_negotiation.cc:206] Beginning negotiation
0929 22:50:39.974749 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 22:50:39.975288 (+ 539us) server_negotiation.cc:407] Connection header received
0929 22:50:39.975312 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 22:50:39.975313 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 22:50:39.975324 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 22:50:39.975372 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0929 22:50:39.976125 (+ 753us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:50:39.976887 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:50:39.977926 (+ 1039us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:50:39.978222 (+ 296us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:50:39.978271 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 22:50:39.982654 (+ 4383us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 22:50:39.982655 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 22:50:39.982656 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 22:50:39.982663 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0929 22:50:39.986244 (+ 3581us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 22:50:39.987554 (+ 1310us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 22:50:39.987555 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 22:50:39.987555 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 22:50:39.987612 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 22:50:39.987823 (+ 211us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 22:50:39.987824 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 22:50:39.987824 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 22:50:39.987951 (+ 127us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 22:50:39.988044 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0929 22:50:39.988263 (+ 219us) server_negotiation.cc:299] Negotiation successful
0929 22:50:39.988346 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84,"thread_start_us":63,"threads_started":1}
I20260929 22:50:39.988756 22270 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:50:39.973787 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53142 (local address 127.14.147.254:45601)
0929 22:50:39.974898 (+ 1111us) server_negotiation.cc:206] Beginning negotiation
0929 22:50:39.974900 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 22:50:39.974907 (+ 7us) server_negotiation.cc:407] Connection header received
0929 22:50:39.974929 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 22:50:39.974930 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 22:50:39.974941 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 22:50:39.975016 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0929 22:50:39.976578 (+ 1562us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:50:39.977371 (+ 793us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:50:39.978323 (+ 952us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:50:39.978554 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:50:39.978605 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 22:50:39.980470 (+ 1865us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 22:50:39.980471 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 22:50:39.980471 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 22:50:39.980478 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0929 22:50:39.986926 (+ 6448us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 22:50:39.987524 (+ 598us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 22:50:39.987526 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 22:50:39.987526 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 22:50:39.987588 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 22:50:39.987794 (+ 206us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 22:50:39.987794 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 22:50:39.987795 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 22:50:39.987915 (+ 120us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 22:50:39.988011 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0929 22:50:39.988409 (+ 398us) server_negotiation.cc:299] Negotiation successful
0929 22:50:39.988488 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1085,"thread_start_us":84,"threads_started":1}
I20260929 22:50:39.990281 21014 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53142:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 22:50:39.991909 21014 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 22:50:39.998978 22284 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:50:39.995524 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.147.193:44967 (local address 127.0.0.1:53482)
0929 22:50:39.996055 (+ 531us) negotiation.cc:107] Waiting for socket to connect
0929 22:50:39.996064 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0929 22:50:39.996117 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 22:50:39.996312 (+ 195us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 22:50:39.996312 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 22:50:39.996315 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 22:50:39.996362 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 22:50:39.996363 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 22:50:39.997224 (+ 861us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 22:50:39.997225 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 22:50:39.998037 (+ 812us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 22:50:39.998039 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 22:50:39.998733 (+ 694us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 22:50:39.998733 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 22:50:39.998787 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 22:50:39.998793 (+ 6us) client_negotiation.cc:769] Sending connection context
0929 22:50:39.998849 (+ 56us) client_negotiation.cc:240] Negotiation successful
0929 22:50:39.998890 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":514,"mutex_wait_us":406,"thread_start_us":76,"threads_started":1}
I20260929 22:50:39.998992 22281 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:50:39.995666 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53482 (local address 127.14.147.193:44967)
0929 22:50:39.995832 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0929 22:50:39.995837 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 22:50:39.996181 (+ 344us) server_negotiation.cc:407] Connection header received
0929 22:50:39.996227 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 22:50:39.996230 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 22:50:39.996246 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 22:50:39.996315 (+ 69us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 22:50:39.996412 (+ 97us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:50:39.997140 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:50:39.998124 (+ 984us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:50:39.998645 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:50:39.998729 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 22:50:39.998771 (+ 42us) server_negotiation.cc:1035] Waiting for connection context
0929 22:50:39.998865 (+ 94us) server_negotiation.cc:299] Negotiation successful
0929 22:50:39.998906 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":129,"thread_start_us":96,"threads_started":1}
I20260929 22:50:39.999825 21919 tablet_service.cc:1467] Processing CreateTablet for tablet cb0677c1d0244de99330a479fcaea225 (DEFAULT_TABLE table=test-table [id=a790b6bc6a724efe9662ebcf67427ca8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 22:50:40.000193 21576 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:50:39.995525 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.147.194:45475 (local address 127.0.0.1:41222)
0929 22:50:39.995648 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0929 22:50:39.995660 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0929 22:50:39.995701 (+ 41us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 22:50:39.996117 (+ 416us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 22:50:39.996118 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 22:50:39.996120 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 22:50:39.996172 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 22:50:39.996174 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 22:50:39.998014 (+ 1840us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 22:50:39.998015 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 22:50:39.998733 (+ 718us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 22:50:39.998735 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 22:50:39.999984 (+ 1249us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 22:50:39.999985 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 22:50:40.000041 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 22:50:40.000045 (+ 4us) client_negotiation.cc:769] Sending connection context
0929 22:50:40.000082 (+ 37us) client_negotiation.cc:240] Negotiation successful
0929 22:50:40.000122 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":103,"mutex_wait_us":18}
I20260929 22:50:40.000227 21919 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cb0677c1d0244de99330a479fcaea225. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 22:50:40.000898 22282 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:50:39.995784 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41222 (local address 127.14.147.194:45475)
0929 22:50:39.995938 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0929 22:50:39.995942 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 22:50:39.995953 (+ 11us) server_negotiation.cc:407] Connection header received
0929 22:50:39.995995 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 22:50:39.995997 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 22:50:39.996013 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 22:50:39.996274 (+ 261us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 22:50:39.996335 (+ 61us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:50:39.997918 (+ 1583us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:50:39.998818 (+ 900us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:50:39.999903 (+ 1085us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:50:40.000673 (+ 770us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 22:50:40.000741 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0929 22:50:40.000770 (+ 29us) server_negotiation.cc:299] Negotiation successful
0929 22:50:40.000816 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":123,"thread_start_us":96,"threads_started":1}
I20260929 22:50:40.000972 22283 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:50:39.995850 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57278 (local address 127.14.147.195:38905)
0929 22:50:39.996277 (+ 427us) server_negotiation.cc:206] Beginning negotiation
0929 22:50:39.996281 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 22:50:39.996291 (+ 10us) server_negotiation.cc:407] Connection header received
0929 22:50:39.996340 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 22:50:39.996342 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 22:50:39.996356 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 22:50:39.996458 (+ 102us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 22:50:39.996536 (+ 78us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:50:39.997847 (+ 1311us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:50:39.999867 (+ 2020us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:50:40.000637 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:50:40.000704 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 22:50:40.000758 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0929 22:50:40.000856 (+ 98us) server_negotiation.cc:299] Negotiation successful
0929 22:50:40.000899 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":374,"thread_start_us":69,"threads_started":1}
I20260929 22:50:40.000931 22280 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:50:39.995552 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.147.195:38905 (local address 127.0.0.1:57278)
0929 22:50:39.995683 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0929 22:50:39.995691 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0929 22:50:39.995730 (+ 39us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 22:50:39.996455 (+ 725us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 22:50:39.996456 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 22:50:39.996458 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 22:50:39.996488 (+ 30us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 22:50:39.996488 (+ 0us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 22:50:39.997962 (+ 1474us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 22:50:39.997963 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 22:50:39.999788 (+ 1825us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 22:50:39.999791 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 22:50:40.000711 (+ 920us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 22:50:40.000712 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 22:50:40.000773 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 22:50:40.000777 (+ 4us) client_negotiation.cc:769] Sending connection context
0929 22:50:40.000826 (+ 49us) client_negotiation.cc:240] Negotiation successful
0929 22:50:40.000867 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":88,"threads_started":1}
I20260929 22:50:40.001673 22056 tablet_service.cc:1467] Processing CreateTablet for tablet cb0677c1d0244de99330a479fcaea225 (DEFAULT_TABLE table=test-table [id=a790b6bc6a724efe9662ebcf67427ca8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 22:50:40.001662 22193 tablet_service.cc:1467] Processing CreateTablet for tablet cb0677c1d0244de99330a479fcaea225 (DEFAULT_TABLE table=test-table [id=a790b6bc6a724efe9662ebcf67427ca8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 22:50:40.002000 22056 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cb0677c1d0244de99330a479fcaea225. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 22:50:40.002000 22193 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cb0677c1d0244de99330a479fcaea225. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 22:50:40.005447 22286 tablet_bootstrap.cc:492] T cb0677c1d0244de99330a479fcaea225 P f9ed52b3ef574d3d8c17014dbf842a54: Bootstrap starting.
I20260929 22:50:40.005569 22285 tablet_bootstrap.cc:492] T cb0677c1d0244de99330a479fcaea225 P e0d74c61684f40338bc384332f90e9c7: Bootstrap starting.
I20260929 22:50:40.006161 22285 tablet_bootstrap.cc:654] T cb0677c1d0244de99330a479fcaea225 P e0d74c61684f40338bc384332f90e9c7: Neither blocks nor log segments found. Creating new log.
I20260929 22:50:40.006165 22286 tablet_bootstrap.cc:654] T cb0677c1d0244de99330a479fcaea225 P f9ed52b3ef574d3d8c17014dbf842a54: Neither blocks nor log segments found. Creating new log.
I20260929 22:50:40.006435 22285 log.cc:824] T cb0677c1d0244de99330a479fcaea225 P e0d74c61684f40338bc384332f90e9c7: Log is configured to *not* fsync() on all Append() calls
I20260929 22:50:40.006477 22286 log.cc:824] T cb0677c1d0244de99330a479fcaea225 P f9ed52b3ef574d3d8c17014dbf842a54: Log is configured to *not* fsync() on all Append() calls
I20260929 22:50:40.006868 22287 tablet_bootstrap.cc:492] T cb0677c1d0244de99330a479fcaea225 P ab4a15db2c3f422e8646d4b5255cf8df: Bootstrap starting.
I20260929 22:50:40.007369 22286 tablet_bootstrap.cc:492] T cb0677c1d0244de99330a479fcaea225 P f9ed52b3ef574d3d8c17014dbf842a54: No bootstrap required, opened a new log
I20260929 22:50:40.007369 22285 tablet_bootstrap.cc:492] T cb0677c1d0244de99330a479fcaea225 P e0d74c61684f40338bc384332f90e9c7: No bootstrap required, opened a new log
I20260929 22:50:40.007447 22287 tablet_bootstrap.cc:654] T cb0677c1d0244de99330a479fcaea225 P ab4a15db2c3f422e8646d4b5255cf8df: Neither blocks nor log segments found. Creating new log.
I20260929 22:50:40.007469 22286 ts_tablet_manager.cc:1397] T cb0677c1d0244de99330a479fcaea225 P f9ed52b3ef574d3d8c17014dbf842a54: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260929 22:50:40.007469 22285 ts_tablet_manager.cc:1397] T cb0677c1d0244de99330a479fcaea225 P e0d74c61684f40338bc384332f90e9c7: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260929 22:50:40.007715 22287 log.cc:824] T cb0677c1d0244de99330a479fcaea225 P ab4a15db2c3f422e8646d4b5255cf8df: Log is configured to *not* fsync() on all Append() calls
I20260929 22:50:40.008338 22287 tablet_bootstrap.cc:492] T cb0677c1d0244de99330a479fcaea225 P ab4a15db2c3f422e8646d4b5255cf8df: No bootstrap required, opened a new log
I20260929 22:50:40.008396 22287 ts_tablet_manager.cc:1397] T cb0677c1d0244de99330a479fcaea225 P ab4a15db2c3f422e8646d4b5255cf8df: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260929 22:50:40.009550 22285 raft_consensus.cc:348] T cb0677c1d0244de99330a479fcaea225 P e0d74c61684f40338bc384332f90e9c7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0d74c61684f40338bc384332f90e9c7" member_type: VOTER last_known_addr { host: "127.14.147.193" port: 44967 } } peers { permanent_uuid: "ab4a15db2c3f422e8646d4b5255cf8df" member_type: VOTER last_known_addr { host: "127.14.147.194" port: 45475 } } peers { permanent_uuid: "f9ed52b3ef574d3d8c17014dbf842a54" member_type: VOTER last_known_addr { host: "127.14.147.195" port: 38905 } }
I20260929 22:50:40.009730 22285 raft_consensus.cc:374] T cb0677c1d0244de99330a479fcaea225 P e0d74c61684f40338bc384332f90e9c7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 22:50:40.009764 22285 raft_consensus.cc:729] T cb0677c1d0244de99330a479fcaea225 P e0d74c61684f40338bc384332f90e9c7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e0d74c61684f40338bc384332f90e9c7, State: Initialized, Role: FOLLOWER
I20260929 22:50:40.009760 22287 raft_consensus.cc:348] T cb0677c1d0244de99330a479fcaea225 P ab4a15db2c3f422e8646d4b5255cf8df [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0d74c61684f40338bc384332f90e9c7" member_type: VOTER last_known_addr { host: "127.14.147.193" port: 44967 } } peers { permanent_uuid: "ab4a15db2c3f422e8646d4b5255cf8df" member_type: VOTER last_known_addr { host: "127.14.147.194" port: 45475 } } peers { permanent_uuid: "f9ed52b3ef574d3d8c17014dbf842a54" member_type: VOTER last_known_addr { host: "127.14.147.195" port: 38905 } }
I20260929 22:50:40.009917 22287 raft_consensus.cc:374] T cb0677c1d0244de99330a479fcaea225 P ab4a15db2c3f422e8646d4b5255cf8df [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 22:50:40.009951 22287 raft_consensus.cc:729] T cb0677c1d0244de99330a479fcaea225 P ab4a15db2c3f422e8646d4b5255cf8df [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ab4a15db2c3f422e8646d4b5255cf8df, State: Initialized, Role: FOLLOWER
I20260929 22:50:40.009922 22285 consensus_queue.cc:260] T cb0677c1d0244de99330a479fcaea225 P e0d74c61684f40338bc384332f90e9c7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0d74c61684f40338bc384332f90e9c7" member_type: VOTER last_known_addr { host: "127.14.147.193" port: 44967 } } peers { permanent_uuid: "ab4a15db2c3f422e8646d4b5255cf8df" member_type: VOTER last_known_addr { host: "127.14.147.194" port: 45475 } } peers { permanent_uuid: "f9ed52b3ef574d3d8c17014dbf842a54" member_type: VOTER last_known_addr { host: "127.14.147.195" port: 38905 } }
I20260929 22:50:40.009923 22286 raft_consensus.cc:348] T cb0677c1d0244de99330a479fcaea225 P f9ed52b3ef574d3d8c17014dbf842a54 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0d74c61684f40338bc384332f90e9c7" member_type: VOTER last_known_addr { host: "127.14.147.193" port: 44967 } } peers { permanent_uuid: "ab4a15db2c3f422e8646d4b5255cf8df" member_type: VOTER last_known_addr { host: "127.14.147.194" port: 45475 } } peers { permanent_uuid: "f9ed52b3ef574d3d8c17014dbf842a54" member_type: VOTER last_known_addr { host: "127.14.147.195" port: 38905 } }
I20260929 22:50:40.010049 22286 raft_consensus.cc:374] T cb0677c1d0244de99330a479fcaea225 P f9ed52b3ef574d3d8c17014dbf842a54 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 22:50:40.010089 22286 raft_consensus.cc:729] T cb0677c1d0244de99330a479fcaea225 P f9ed52b3ef574d3d8c17014dbf842a54 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f9ed52b3ef574d3d8c17014dbf842a54, State: Initialized, Role: FOLLOWER
I20260929 22:50:40.010073 22287 consensus_queue.cc:260] T cb0677c1d0244de99330a479fcaea225 P ab4a15db2c3f422e8646d4b5255cf8df [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0d74c61684f40338bc384332f90e9c7" member_type: VOTER last_known_addr { host: "127.14.147.193" port: 44967 } } peers { permanent_uuid: "ab4a15db2c3f422e8646d4b5255cf8df" member_type: VOTER last_known_addr { host: "127.14.147.194" port: 45475 } } peers { permanent_uuid: "f9ed52b3ef574d3d8c17014dbf842a54" member_type: VOTER last_known_addr { host: "127.14.147.195" port: 38905 } }
I20260929 22:50:40.010250 22285 ts_tablet_manager.cc:1428] T cb0677c1d0244de99330a479fcaea225 P e0d74c61684f40338bc384332f90e9c7: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260929 22:50:40.010221 22286 consensus_queue.cc:260] T cb0677c1d0244de99330a479fcaea225 P f9ed52b3ef574d3d8c17014dbf842a54 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0d74c61684f40338bc384332f90e9c7" member_type: VOTER last_known_addr { host: "127.14.147.193" port: 44967 } } peers { permanent_uuid: "ab4a15db2c3f422e8646d4b5255cf8df" member_type: VOTER last_known_addr { host: "127.14.147.194" port: 45475 } } peers { permanent_uuid: "f9ed52b3ef574d3d8c17014dbf842a54" member_type: VOTER last_known_addr { host: "127.14.147.195" port: 38905 } }
I20260929 22:50:40.010336 22287 ts_tablet_manager.cc:1428] T cb0677c1d0244de99330a479fcaea225 P ab4a15db2c3f422e8646d4b5255cf8df: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260929 22:50:40.010449 22123 heartbeater.cc:503] Master 127.14.147.254:45601 was elected leader, sending a full tablet report...
I20260929 22:50:40.010462 22286 ts_tablet_manager.cc:1428] T cb0677c1d0244de99330a479fcaea225 P f9ed52b3ef574d3d8c17014dbf842a54: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260929 22:50:40.011467 22260 heartbeater.cc:503] Master 127.14.147.254:45601 was elected leader, sending a full tablet report...
I20260929 22:50:40.011924 21986 heartbeater.cc:503] Master 127.14.147.254:45601 was elected leader, sending a full tablet report...
I20260929 22:50:40.027458 22293 raft_consensus.cc:482] T cb0677c1d0244de99330a479fcaea225 P f9ed52b3ef574d3d8c17014dbf842a54 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 22:50:40.027596 22293 raft_consensus.cc:504] T cb0677c1d0244de99330a479fcaea225 P f9ed52b3ef574d3d8c17014dbf842a54 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0d74c61684f40338bc384332f90e9c7" member_type: VOTER last_known_addr { host: "127.14.147.193" port: 44967 } } peers { permanent_uuid: "ab4a15db2c3f422e8646d4b5255cf8df" member_type: VOTER last_known_addr { host: "127.14.147.194" port: 45475 } } peers { permanent_uuid: "f9ed52b3ef574d3d8c17014dbf842a54" member_type: VOTER last_known_addr { host: "127.14.147.195" port: 38905 } }
I20260929 22:50:40.027980 22293 leader_election.cc:288] T cb0677c1d0244de99330a479fcaea225 P f9ed52b3ef574d3d8c17014dbf842a54 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e0d74c61684f40338bc384332f90e9c7 (127.14.147.193:44967), ab4a15db2c3f422e8646d4b5255cf8df (127.14.147.194:45475)
I20260929 22:50:40.031917 22281 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:50:40.028256 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.147.195:54015 (local address 127.14.147.193:44967)
0929 22:50:40.028301 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0929 22:50:40.028303 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 22:50:40.028314 (+ 11us) server_negotiation.cc:407] Connection header received
0929 22:50:40.028374 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 22:50:40.028375 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 22:50:40.028386 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 22:50:40.028441 (+ 55us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 22:50:40.028625 (+ 184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:50:40.029382 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:50:40.030936 (+ 1554us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:50:40.031550 (+ 614us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:50:40.031629 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 22:50:40.031639 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0929 22:50:40.031797 (+ 158us) server_negotiation.cc:299] Negotiation successful
0929 22:50:40.031849 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260929 22:50:40.031916 22282 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:50:40.028327 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.147.195:47133 (local address 127.14.147.194:45475)
0929 22:50:40.028366 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0929 22:50:40.028368 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 22:50:40.028376 (+ 8us) server_negotiation.cc:407] Connection header received
0929 22:50:40.028399 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 22:50:40.028400 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 22:50:40.028407 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 22:50:40.028458 (+ 51us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 22:50:40.028627 (+ 169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:50:40.029389 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:50:40.030941 (+ 1552us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:50:40.031550 (+ 609us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:50:40.031629 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 22:50:40.031639 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0929 22:50:40.031797 (+ 158us) server_negotiation.cc:299] Negotiation successful
0929 22:50:40.031843 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260929 22:50:40.031929 22264 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:50:40.028199 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.147.194:45475 (local address 127.14.147.195:47133)
0929 22:50:40.028282 (+ 83us) negotiation.cc:107] Waiting for socket to connect
0929 22:50:40.028291 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0929 22:50:40.028323 (+ 32us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 22:50:40.028465 (+ 142us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 22:50:40.028466 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 22:50:40.028467 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 22:50:40.028564 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 22:50:40.028566 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 22:50:40.029499 (+ 933us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 22:50:40.029501 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 22:50:40.030814 (+ 1313us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 22:50:40.030818 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 22:50:40.031632 (+ 814us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 22:50:40.031633 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 22:50:40.031728 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 22:50:40.031730 (+ 2us) client_negotiation.cc:769] Sending connection context
0929 22:50:40.031786 (+ 56us) client_negotiation.cc:240] Negotiation successful
0929 22:50:40.031831 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":73,"mutex_wait_us":87}
I20260929 22:50:40.031929 22263 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:50:40.028198 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.147.193:44967 (local address 127.14.147.195:54015)
0929 22:50:40.028245 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0929 22:50:40.028257 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0929 22:50:40.028303 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 22:50:40.028462 (+ 159us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 22:50:40.028462 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 22:50:40.028465 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 22:50:40.028564 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 22:50:40.028566 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 22:50:40.029499 (+ 933us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 22:50:40.029501 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 22:50:40.030814 (+ 1313us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 22:50:40.030818 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 22:50:40.031632 (+ 814us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 22:50:40.031633 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 22:50:40.031720 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 22:50:40.031725 (+ 5us) client_negotiation.cc:769] Sending connection context
0929 22:50:40.031786 (+ 61us) client_negotiation.cc:240] Negotiation successful
0929 22:50:40.031831 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":27,"mutex_wait_us":101}
I20260929 22:50:40.032366 21939 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cb0677c1d0244de99330a479fcaea225" candidate_uuid: "f9ed52b3ef574d3d8c17014dbf842a54" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e0d74c61684f40338bc384332f90e9c7" is_pre_election: true
I20260929 22:50:40.032531 21939 raft_consensus.cc:2454] T cb0677c1d0244de99330a479fcaea225 P e0d74c61684f40338bc384332f90e9c7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f9ed52b3ef574d3d8c17014dbf842a54 in term 0.
I20260929 22:50:40.032703 22076 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cb0677c1d0244de99330a479fcaea225" candidate_uuid: "f9ed52b3ef574d3d8c17014dbf842a54" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ab4a15db2c3f422e8646d4b5255cf8df" is_pre_election: true
I20260929 22:50:40.032745 22149 leader_election.cc:302] T cb0677c1d0244de99330a479fcaea225 P f9ed52b3ef574d3d8c17014dbf842a54 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e0d74c61684f40338bc384332f90e9c7, f9ed52b3ef574d3d8c17014dbf842a54; no voters:
I20260929 22:50:40.032830 22076 raft_consensus.cc:2454] T cb0677c1d0244de99330a479fcaea225 P ab4a15db2c3f422e8646d4b5255cf8df [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f9ed52b3ef574d3d8c17014dbf842a54 in term 0.
I20260929 22:50:40.032867 22293 raft_consensus.cc:2781] T cb0677c1d0244de99330a479fcaea225 P f9ed52b3ef574d3d8c17014dbf842a54 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 22:50:40.032948 22293 raft_consensus.cc:482] T cb0677c1d0244de99330a479fcaea225 P f9ed52b3ef574d3d8c17014dbf842a54 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 22:50:40.032963 22293 raft_consensus.cc:3037] T cb0677c1d0244de99330a479fcaea225 P f9ed52b3ef574d3d8c17014dbf842a54 [term 0 FOLLOWER]: Advancing to term 1
I20260929 22:50:40.033676 22293 raft_consensus.cc:504] T cb0677c1d0244de99330a479fcaea225 P f9ed52b3ef574d3d8c17014dbf842a54 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0d74c61684f40338bc384332f90e9c7" member_type: VOTER last_known_addr { host: "127.14.147.193" port: 44967 } } peers { permanent_uuid: "ab4a15db2c3f422e8646d4b5255cf8df" member_type: VOTER last_known_addr { host: "127.14.147.194" port: 45475 } } peers { permanent_uuid: "f9ed52b3ef574d3d8c17014dbf842a54" member_type: VOTER last_known_addr { host: "127.14.147.195" port: 38905 } }
I20260929 22:50:40.033802 22293 leader_election.cc:288] T cb0677c1d0244de99330a479fcaea225 P f9ed52b3ef574d3d8c17014dbf842a54 [CANDIDATE]: Term 1 election: Requested vote from peers e0d74c61684f40338bc384332f90e9c7 (127.14.147.193:44967), ab4a15db2c3f422e8646d4b5255cf8df (127.14.147.194:45475)
I20260929 22:50:40.033958 21939 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cb0677c1d0244de99330a479fcaea225" candidate_uuid: "f9ed52b3ef574d3d8c17014dbf842a54" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e0d74c61684f40338bc384332f90e9c7"
I20260929 22:50:40.034003 22076 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cb0677c1d0244de99330a479fcaea225" candidate_uuid: "f9ed52b3ef574d3d8c17014dbf842a54" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ab4a15db2c3f422e8646d4b5255cf8df"
I20260929 22:50:40.034039 21939 raft_consensus.cc:3037] T cb0677c1d0244de99330a479fcaea225 P e0d74c61684f40338bc384332f90e9c7 [term 0 FOLLOWER]: Advancing to term 1
I20260929 22:50:40.034072 22076 raft_consensus.cc:3037] T cb0677c1d0244de99330a479fcaea225 P ab4a15db2c3f422e8646d4b5255cf8df [term 0 FOLLOWER]: Advancing to term 1
I20260929 22:50:40.034920 22076 raft_consensus.cc:2454] T cb0677c1d0244de99330a479fcaea225 P ab4a15db2c3f422e8646d4b5255cf8df [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f9ed52b3ef574d3d8c17014dbf842a54 in term 1.
I20260929 22:50:40.034920 21939 raft_consensus.cc:2454] T cb0677c1d0244de99330a479fcaea225 P e0d74c61684f40338bc384332f90e9c7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f9ed52b3ef574d3d8c17014dbf842a54 in term 1.
I20260929 22:50:40.035096 22149 leader_election.cc:302] T cb0677c1d0244de99330a479fcaea225 P f9ed52b3ef574d3d8c17014dbf842a54 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e0d74c61684f40338bc384332f90e9c7, f9ed52b3ef574d3d8c17014dbf842a54; no voters:
I20260929 22:50:40.035177 22293 raft_consensus.cc:2781] T cb0677c1d0244de99330a479fcaea225 P f9ed52b3ef574d3d8c17014dbf842a54 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 22:50:40.035328 22293 raft_consensus.cc:686] T cb0677c1d0244de99330a479fcaea225 P f9ed52b3ef574d3d8c17014dbf842a54 [term 1 LEADER]: Becoming Leader. State: Replica: f9ed52b3ef574d3d8c17014dbf842a54, State: Running, Role: LEADER
I20260929 22:50:40.035439 22293 consensus_queue.cc:237] T cb0677c1d0244de99330a479fcaea225 P f9ed52b3ef574d3d8c17014dbf842a54 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0d74c61684f40338bc384332f90e9c7" member_type: VOTER last_known_addr { host: "127.14.147.193" port: 44967 } } peers { permanent_uuid: "ab4a15db2c3f422e8646d4b5255cf8df" member_type: VOTER last_known_addr { host: "127.14.147.194" port: 45475 } } peers { permanent_uuid: "f9ed52b3ef574d3d8c17014dbf842a54" member_type: VOTER last_known_addr { host: "127.14.147.195" port: 38905 } }
I20260929 22:50:40.036407 21014 catalog_manager.cc:5495] T cb0677c1d0244de99330a479fcaea225 P f9ed52b3ef574d3d8c17014dbf842a54 reported cstate change: term changed from 0 to 1, leader changed from <none> to f9ed52b3ef574d3d8c17014dbf842a54 (127.14.147.195). New cstate: current_term: 1 leader_uuid: "f9ed52b3ef574d3d8c17014dbf842a54" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0d74c61684f40338bc384332f90e9c7" member_type: VOTER last_known_addr { host: "127.14.147.193" port: 44967 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ab4a15db2c3f422e8646d4b5255cf8df" member_type: VOTER last_known_addr { host: "127.14.147.194" port: 45475 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f9ed52b3ef574d3d8c17014dbf842a54" member_type: VOTER last_known_addr { host: "127.14.147.195" port: 38905 } health_report { overall_health: HEALTHY } } }
I20260929 22:50:40.045379 22283 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:50:40.043060 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57294 (local address 127.14.147.195:38905)
0929 22:50:40.043117 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0929 22:50:40.043119 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 22:50:40.043127 (+ 8us) server_negotiation.cc:407] Connection header received
0929 22:50:40.043159 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 22:50:40.043159 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 22:50:40.043179 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 22:50:40.043239 (+ 60us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 22:50:40.043378 (+ 139us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:50:40.043886 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:50:40.044671 (+ 785us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:50:40.044918 (+ 247us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:50:40.044972 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 22:50:40.045097 (+ 125us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 22:50:40.045201 (+ 104us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 22:50:40.045258 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0929 22:50:40.045271 (+ 13us) server_negotiation.cc:299] Negotiation successful
0929 22:50:40.045289 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260929 22:50:40.047458 21939 raft_consensus.cc:1261] T cb0677c1d0244de99330a479fcaea225 P e0d74c61684f40338bc384332f90e9c7 [term 1 FOLLOWER]: Refusing update from remote peer f9ed52b3ef574d3d8c17014dbf842a54: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 22:50:40.047672 22076 raft_consensus.cc:1261] T cb0677c1d0244de99330a479fcaea225 P ab4a15db2c3f422e8646d4b5255cf8df [term 1 FOLLOWER]: Refusing update from remote peer f9ed52b3ef574d3d8c17014dbf842a54: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 22:50:40.047791 22293 consensus_queue.cc:1035] T cb0677c1d0244de99330a479fcaea225 P f9ed52b3ef574d3d8c17014dbf842a54 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e0d74c61684f40338bc384332f90e9c7" member_type: VOTER last_known_addr { host: "127.14.147.193" port: 44967 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 22:50:40.047966 22294 consensus_queue.cc:1035] T cb0677c1d0244de99330a479fcaea225 P f9ed52b3ef574d3d8c17014dbf842a54 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab4a15db2c3f422e8646d4b5255cf8df" member_type: VOTER last_known_addr { host: "127.14.147.194" port: 45475 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 22:50:40.049575 22280 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:50:40.046326 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.147.252:41485 (local address 127.0.0.1:60062)
0929 22:50:40.046402 (+ 76us) negotiation.cc:107] Waiting for socket to connect
0929 22:50:40.046415 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 22:50:40.046460 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 22:50:40.046724 (+ 264us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 22:50:40.046725 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 22:50:40.046728 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 22:50:40.046776 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 22:50:40.046778 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 22:50:40.047502 (+ 724us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 22:50:40.047503 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 22:50:40.048537 (+ 1034us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 22:50:40.048541 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 22:50:40.049323 (+ 782us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 22:50:40.049324 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 22:50:40.049395 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 22:50:40.049399 (+ 4us) client_negotiation.cc:769] Sending connection context
0929 22:50:40.049447 (+ 48us) client_negotiation.cc:240] Negotiation successful
0929 22:50:40.049502 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":51}
I20260929 22:50:40.050590 22297 mvcc.cc:205] Tried to move back new op lower bound from 7334798295231954944 to 7334798295185915904. Current Snapshot: MvccSnapshot[applied={T|T < 7334798295231954944}]
I20260929 22:50:40.051851 22298 mvcc.cc:205] Tried to move back new op lower bound from 7334798295231954944 to 7334798295185915904. Current Snapshot: MvccSnapshot[applied={T|T < 7334798295231954944 or (T in {7334798295231954944})}]
I20260929 22:50:40.052481 22272 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:50:40.046500 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60062 (local address 127.14.147.252:41485)
0929 22:50:40.046558 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0929 22:50:40.046560 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 22:50:40.046567 (+ 7us) server_negotiation.cc:407] Connection header received
0929 22:50:40.046591 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 22:50:40.046592 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 22:50:40.046611 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 22:50:40.046661 (+ 50us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 22:50:40.046854 (+ 193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:50:40.047410 (+ 556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:50:40.048638 (+ 1228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:50:40.049218 (+ 580us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:50:40.049293 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 22:50:40.049372 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0929 22:50:40.052327 (+ 2955us) server_negotiation.cc:299] Negotiation successful
0929 22:50:40.052410 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260929 22:50:40.052675 22299 mvcc.cc:205] Tried to move back new op lower bound from 7334798295231954944 to 7334798295185915904. Current Snapshot: MvccSnapshot[applied={T|T < 7334798295231954944 or (T in {7334798295231954944})}]
I20260929 22:50:40.052762 21815 raft_consensus.cc:1203] T 00000000000000000000000000000000 P 98a8b44ee91645f8a3dc6f02d1c5c698 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.10] Dedup: 1.7->[1.8-1.10]
I20260929 22:50:40.053627 21844 sys_catalog.cc:455] T 00000000000000000000000000000000 P 98a8b44ee91645f8a3dc6f02d1c5c698 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d38d0f6f0d3c4d2b8b7131e3f0f72ffe. Latest consensus state: current_term: 1 leader_uuid: "d38d0f6f0d3c4d2b8b7131e3f0f72ffe" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "d38d0f6f0d3c4d2b8b7131e3f0f72ffe" member_type: VOTER last_known_addr { host: "127.14.147.254" port: 45601 } } peers { permanent_uuid: "e820b147283e4503b216af02b09dce4a" member_type: VOTER last_known_addr { host: "127.14.147.253" port: 40121 } attrs { promote: false } } peers { permanent_uuid: "98a8b44ee91645f8a3dc6f02d1c5c698" member_type: NON_VOTER last_known_addr { host: "127.14.147.252" port: 41485 } attrs { promote: true } } }
I20260929 22:50:40.053702 21844 sys_catalog.cc:458] T 00000000000000000000000000000000 P 98a8b44ee91645f8a3dc6f02d1c5c698 [sys.catalog]: This master's current role is: LEARNER
I20260929 22:50:40.054286 21014 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53142:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 22:50:40.054379 21014 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53142:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 22:50:40.054881 21768 sys_catalog.cc:455] T 00000000000000000000000000000000 P d38d0f6f0d3c4d2b8b7131e3f0f72ffe [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "d38d0f6f0d3c4d2b8b7131e3f0f72ffe" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "d38d0f6f0d3c4d2b8b7131e3f0f72ffe" member_type: VOTER last_known_addr { host: "127.14.147.254" port: 45601 } } peers { permanent_uuid: "e820b147283e4503b216af02b09dce4a" member_type: VOTER last_known_addr { host: "127.14.147.253" port: 40121 } attrs { promote: false } } peers { permanent_uuid: "98a8b44ee91645f8a3dc6f02d1c5c698" member_type: NON_VOTER last_known_addr { host: "127.14.147.252" port: 41485 } attrs { promote: true } } }
I20260929 22:50:40.054981 21768 sys_catalog.cc:458] T 00000000000000000000000000000000 P d38d0f6f0d3c4d2b8b7131e3f0f72ffe [sys.catalog]: This master's current role is: LEADER
I20260929 22:50:40.055593 21769 raft_consensus.cc:1050] T 00000000000000000000000000000000 P d38d0f6f0d3c4d2b8b7131e3f0f72ffe: attempting to promote NON_VOTER 98a8b44ee91645f8a3dc6f02d1c5c698 to VOTER
I20260929 22:50:40.055733 21769 consensus_queue.cc:237] T 00000000000000000000000000000000 P d38d0f6f0d3c4d2b8b7131e3f0f72ffe [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 1.11, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 12 OBSOLETE_local: true peers { permanent_uuid: "d38d0f6f0d3c4d2b8b7131e3f0f72ffe" member_type: VOTER last_known_addr { host: "127.14.147.254" port: 45601 } } peers { permanent_uuid: "e820b147283e4503b216af02b09dce4a" member_type: VOTER last_known_addr { host: "127.14.147.253" port: 40121 } attrs { promote: false } } peers { permanent_uuid: "98a8b44ee91645f8a3dc6f02d1c5c698" member_type: VOTER last_known_addr { host: "127.14.147.252" port: 41485 } attrs { promote: false } }
I20260929 22:50:40.056125 21815 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 98a8b44ee91645f8a3dc6f02d1c5c698 [term 1 LEARNER]: Refusing update from remote peer d38d0f6f0d3c4d2b8b7131e3f0f72ffe: Log matching property violated. Preceding OpId in replica: term: 1 index: 11. Preceding OpId from leader: term: 1 index: 12. (index mismatch)
I20260929 22:50:40.056247 21627 raft_consensus.cc:1261] T 00000000000000000000000000000000 P e820b147283e4503b216af02b09dce4a [term 1 FOLLOWER]: Refusing update from remote peer d38d0f6f0d3c4d2b8b7131e3f0f72ffe: Log matching property violated. Preceding OpId in replica: term: 1 index: 11. Preceding OpId from leader: term: 1 index: 12. (index mismatch)
I20260929 22:50:40.056452 21578 consensus_queue.cc:1035] T 00000000000000000000000000000000 P d38d0f6f0d3c4d2b8b7131e3f0f72ffe [LEADER]: Connected to new peer: Peer: permanent_uuid: "98a8b44ee91645f8a3dc6f02d1c5c698" member_type: VOTER last_known_addr { host: "127.14.147.252" port: 41485 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 10, Time since last communication: 0.000s
I20260929 22:50:40.056567 21768 consensus_queue.cc:1035] T 00000000000000000000000000000000 P d38d0f6f0d3c4d2b8b7131e3f0f72ffe [LEADER]: Connected to new peer: Peer: permanent_uuid: "e820b147283e4503b216af02b09dce4a" member_type: VOTER last_known_addr { host: "127.14.147.253" port: 40121 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 10, Time since last communication: 0.000s
I20260929 22:50:40.057140 21578 raft_consensus.cc:2932] T 00000000000000000000000000000000 P d38d0f6f0d3c4d2b8b7131e3f0f72ffe [term 1 LEADER]: Committing config change with OpId 1.12: config changed from index 7 to 12, 98a8b44ee91645f8a3dc6f02d1c5c698 (127.14.147.252) changed from NON_VOTER to VOTER. New config: { opid_index: 12 OBSOLETE_local: true peers { permanent_uuid: "d38d0f6f0d3c4d2b8b7131e3f0f72ffe" member_type: VOTER last_known_addr { host: "127.14.147.254" port: 45601 } } peers { permanent_uuid: "e820b147283e4503b216af02b09dce4a" member_type: VOTER last_known_addr { host: "127.14.147.253" port: 40121 } attrs { promote: false } } peers { permanent_uuid: "98a8b44ee91645f8a3dc6f02d1c5c698" member_type: VOTER last_known_addr { host: "127.14.147.252" port: 41485 } attrs { promote: false } } }
I20260929 22:50:40.057336 21815 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 98a8b44ee91645f8a3dc6f02d1c5c698 [term 1 FOLLOWER]: Committing config change with OpId 1.12: config changed from index 7 to 12, 98a8b44ee91645f8a3dc6f02d1c5c698 (127.14.147.252) changed from NON_VOTER to VOTER. New config: { opid_index: 12 OBSOLETE_local: true peers { permanent_uuid: "d38d0f6f0d3c4d2b8b7131e3f0f72ffe" member_type: VOTER last_known_addr { host: "127.14.147.254" port: 45601 } } peers { permanent_uuid: "e820b147283e4503b216af02b09dce4a" member_type: VOTER last_known_addr { host: "127.14.147.253" port: 40121 } attrs { promote: false } } peers { permanent_uuid: "98a8b44ee91645f8a3dc6f02d1c5c698" member_type: VOTER last_known_addr { host: "127.14.147.252" port: 41485 } attrs { promote: false } } }
I20260929 22:50:40.057376 21627 raft_consensus.cc:2932] T 00000000000000000000000000000000 P e820b147283e4503b216af02b09dce4a [term 1 FOLLOWER]: Committing config change with OpId 1.12: config changed from index 7 to 12, 98a8b44ee91645f8a3dc6f02d1c5c698 (127.14.147.252) changed from NON_VOTER to VOTER. New config: { opid_index: 12 OBSOLETE_local: true peers { permanent_uuid: "d38d0f6f0d3c4d2b8b7131e3f0f72ffe" member_type: VOTER last_known_addr { host: "127.14.147.254" port: 45601 } } peers { permanent_uuid: "e820b147283e4503b216af02b09dce4a" member_type: VOTER last_known_addr { host: "127.14.147.253" port: 40121 } attrs { promote: false } } peers { permanent_uuid: "98a8b44ee91645f8a3dc6f02d1c5c698" member_type: VOTER last_known_addr { host: "127.14.147.252" port: 41485 } attrs { promote: false } } }
I20260929 22:50:40.058136 21578 sys_catalog.cc:455] T 00000000000000000000000000000000 P d38d0f6f0d3c4d2b8b7131e3f0f72ffe [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "d38d0f6f0d3c4d2b8b7131e3f0f72ffe" committed_config { opid_index: 12 OBSOLETE_local: true peers { permanent_uuid: "d38d0f6f0d3c4d2b8b7131e3f0f72ffe" member_type: VOTER last_known_addr { host: "127.14.147.254" port: 45601 } } peers { permanent_uuid: "e820b147283e4503b216af02b09dce4a" member_type: VOTER last_known_addr { host: "127.14.147.253" port: 40121 } attrs { promote: false } } peers { permanent_uuid: "98a8b44ee91645f8a3dc6f02d1c5c698" member_type: VOTER last_known_addr { host: "127.14.147.252" port: 41485 } attrs { promote: false } } }
I20260929 22:50:40.058205 21578 sys_catalog.cc:458] T 00000000000000000000000000000000 P d38d0f6f0d3c4d2b8b7131e3f0f72ffe [sys.catalog]: This master's current role is: LEADER
I20260929 22:50:40.058254 21578 sys_catalog.cc:455] T 00000000000000000000000000000000 P d38d0f6f0d3c4d2b8b7131e3f0f72ffe [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "d38d0f6f0d3c4d2b8b7131e3f0f72ffe" committed_config { opid_index: 12 OBSOLETE_local: true peers { permanent_uuid: "d38d0f6f0d3c4d2b8b7131e3f0f72ffe" member_type: VOTER last_known_addr { host: "127.14.147.254" port: 45601 } } peers { permanent_uuid: "e820b147283e4503b216af02b09dce4a" member_type: VOTER last_known_addr { host: "127.14.147.253" port: 40121 } attrs { promote: false } } peers { permanent_uuid: "98a8b44ee91645f8a3dc6f02d1c5c698" member_type: VOTER last_known_addr { host: "127.14.147.252" port: 41485 } attrs { promote: false } } }
I20260929 22:50:40.058291 21578 sys_catalog.cc:458] T 00000000000000000000000000000000 P d38d0f6f0d3c4d2b8b7131e3f0f72ffe [sys.catalog]: This master's current role is: LEADER
I20260929 22:50:40.058275 21844 sys_catalog.cc:455] T 00000000000000000000000000000000 P 98a8b44ee91645f8a3dc6f02d1c5c698 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "d38d0f6f0d3c4d2b8b7131e3f0f72ffe" committed_config { opid_index: 12 OBSOLETE_local: true peers { permanent_uuid: "d38d0f6f0d3c4d2b8b7131e3f0f72ffe" member_type: VOTER last_known_addr { host: "127.14.147.254" port: 45601 } } peers { permanent_uuid: "e820b147283e4503b216af02b09dce4a" member_type: VOTER last_known_addr { host: "127.14.147.253" port: 40121 } attrs { promote: false } } peers { permanent_uuid: "98a8b44ee91645f8a3dc6f02d1c5c698" member_type: VOTER last_known_addr { host: "127.14.147.252" port: 41485 } attrs { promote: false } } }
I20260929 22:50:40.058323 21578 sys_catalog.cc:455] T 00000000000000000000000000000000 P d38d0f6f0d3c4d2b8b7131e3f0f72ffe [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "d38d0f6f0d3c4d2b8b7131e3f0f72ffe" committed_config { opid_index: 12 OBSOLETE_local: true peers { permanent_uuid: "d38d0f6f0d3c4d2b8b7131e3f0f72ffe" member_type: VOTER last_known_addr { host: "127.14.147.254" port: 45601 } } peers { permanent_uuid: "e820b147283e4503b216af02b09dce4a" member_type: VOTER last_known_addr { host: "127.14.147.253" port: 40121 } attrs { promote: false } } peers { permanent_uuid: "98a8b44ee91645f8a3dc6f02d1c5c698" member_type: VOTER last_known_addr { host: "127.14.147.252" port: 41485 } attrs { promote: false } } }
I20260929 22:50:40.058292 21656 sys_catalog.cc:455] T 00000000000000000000000000000000 P e820b147283e4503b216af02b09dce4a [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "d38d0f6f0d3c4d2b8b7131e3f0f72ffe" committed_config { opid_index: 12 OBSOLETE_local: true peers { permanent_uuid: "d38d0f6f0d3c4d2b8b7131e3f0f72ffe" member_type: VOTER last_known_addr { host: "127.14.147.254" port: 45601 } } peers { permanent_uuid: "e820b147283e4503b216af02b09dce4a" member_type: VOTER last_known_addr { host: "127.14.147.253" port: 40121 } attrs { promote: false } } peers { permanent_uuid: "98a8b44ee91645f8a3dc6f02d1c5c698" member_type: VOTER last_known_addr { host: "127.14.147.252" port: 41485 } attrs { promote: false } } }
I20260929 22:50:40.058349 21844 sys_catalog.cc:458] T 00000000000000000000000000000000 P 98a8b44ee91645f8a3dc6f02d1c5c698 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 22:50:40.058358 21578 sys_catalog.cc:458] T 00000000000000000000000000000000 P d38d0f6f0d3c4d2b8b7131e3f0f72ffe [sys.catalog]: This master's current role is: LEADER
I20260929 22:50:40.058367 21656 sys_catalog.cc:458] T 00000000000000000000000000000000 P e820b147283e4503b216af02b09dce4a [sys.catalog]: This master's current role is: FOLLOWER
I20260929 22:50:40.058689 21014 catalog_manager.cc:5782] T 00000000000000000000000000000000 P d38d0f6f0d3c4d2b8b7131e3f0f72ffe: Sending DeleteTablet for 3 replicas of tablet cb0677c1d0244de99330a479fcaea225
I20260929 22:50:40.059042 22056 tablet_service.cc:1514] Processing DeleteTablet for tablet cb0677c1d0244de99330a479fcaea225 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 22:50:40 UTC) from {username='oryx', principal='oryx/127.14.147.254@KRBTEST.COM'} at 127.0.0.1:41222
I20260929 22:50:40.059042 21919 tablet_service.cc:1514] Processing DeleteTablet for tablet cb0677c1d0244de99330a479fcaea225 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 22:50:40 UTC) from {username='oryx', principal='oryx/127.14.147.254@KRBTEST.COM'} at 127.0.0.1:53482
I20260929 22:50:40.059096 22193 tablet_service.cc:1514] Processing DeleteTablet for tablet cb0677c1d0244de99330a479fcaea225 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 22:50:40 UTC) from {username='oryx', principal='oryx/127.14.147.254@KRBTEST.COM'} at 127.0.0.1:57278
I20260929 22:50:40.059538 22312 tablet_replica.cc:330] T cb0677c1d0244de99330a479fcaea225 P f9ed52b3ef574d3d8c17014dbf842a54: stopping tablet replica
I20260929 22:50:40.059664 22312 raft_consensus.cc:2229] T cb0677c1d0244de99330a479fcaea225 P f9ed52b3ef574d3d8c17014dbf842a54 [term 1 LEADER]: Raft consensus shutting down.
I20260929 22:50:40.059795 22312 raft_consensus.cc:2258] T cb0677c1d0244de99330a479fcaea225 P f9ed52b3ef574d3d8c17014dbf842a54 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 22:50:40.059962 22311 tablet_replica.cc:330] T cb0677c1d0244de99330a479fcaea225 P e0d74c61684f40338bc384332f90e9c7: stopping tablet replica
I20260929 22:50:40.060065 22311 raft_consensus.cc:2229] T cb0677c1d0244de99330a479fcaea225 P e0d74c61684f40338bc384332f90e9c7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 22:50:40.060060 14927 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhWOCLk/build/release/bin/kudu with pid 20982
I20260929 22:50:40.060174 22311 raft_consensus.cc:2258] T cb0677c1d0244de99330a479fcaea225 P e0d74c61684f40338bc384332f90e9c7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 22:50:40.060328 22312 ts_tablet_manager.cc:1905] T cb0677c1d0244de99330a479fcaea225 P f9ed52b3ef574d3d8c17014dbf842a54: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 22:50:40.060628 22310 tablet_replica.cc:330] T cb0677c1d0244de99330a479fcaea225 P ab4a15db2c3f422e8646d4b5255cf8df: stopping tablet replica
I20260929 22:50:40.060756 22310 raft_consensus.cc:2229] T cb0677c1d0244de99330a479fcaea225 P ab4a15db2c3f422e8646d4b5255cf8df [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 22:50:40.060858 22310 raft_consensus.cc:2258] T cb0677c1d0244de99330a479fcaea225 P ab4a15db2c3f422e8646d4b5255cf8df [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 22:50:40.061161 22311 ts_tablet_manager.cc:1905] T cb0677c1d0244de99330a479fcaea225 P e0d74c61684f40338bc384332f90e9c7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 22:50:40.062083 22312 ts_tablet_manager.cc:1918] T cb0677c1d0244de99330a479fcaea225 P f9ed52b3ef574d3d8c17014dbf842a54: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 22:50:40.062170 22312 log.cc:1196] T cb0677c1d0244de99330a479fcaea225 P f9ed52b3ef574d3d8c17014dbf842a54: Deleting WAL directory at /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790722213368738-14927-0/minicluster-data/ts-2/wal/wals/cb0677c1d0244de99330a479fcaea225
I20260929 22:50:40.062418 22312 ts_tablet_manager.cc:1939] T cb0677c1d0244de99330a479fcaea225 P f9ed52b3ef574d3d8c17014dbf842a54: Deleting consensus metadata
I20260929 22:50:40.062721 22311 ts_tablet_manager.cc:1918] T cb0677c1d0244de99330a479fcaea225 P e0d74c61684f40338bc384332f90e9c7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 22:50:40.062796 22311 log.cc:1196] T cb0677c1d0244de99330a479fcaea225 P e0d74c61684f40338bc384332f90e9c7: Deleting WAL directory at /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790722213368738-14927-0/minicluster-data/ts-0/wal/wals/cb0677c1d0244de99330a479fcaea225
I20260929 22:50:40.063037 22311 ts_tablet_manager.cc:1939] T cb0677c1d0244de99330a479fcaea225 P e0d74c61684f40338bc384332f90e9c7: Deleting consensus metadata
I20260929 22:50:40.063796 22310 ts_tablet_manager.cc:1905] T cb0677c1d0244de99330a479fcaea225 P ab4a15db2c3f422e8646d4b5255cf8df: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 22:50:40.065373 22310 ts_tablet_manager.cc:1918] T cb0677c1d0244de99330a479fcaea225 P ab4a15db2c3f422e8646d4b5255cf8df: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
W20260929 22:50:40.065394 22146 connection.cc:535] server connection from 127.0.0.1:57278 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260929 22:50:40.065454 22310 log.cc:1196] T cb0677c1d0244de99330a479fcaea225 P ab4a15db2c3f422e8646d4b5255cf8df: Deleting WAL directory at /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790722213368738-14927-0/minicluster-data/ts-1/wal/wals/cb0677c1d0244de99330a479fcaea225
I20260929 22:50:40.065717 22310 ts_tablet_manager.cc:1939] T cb0677c1d0244de99330a479fcaea225 P ab4a15db2c3f422e8646d4b5255cf8df: Deleting consensus metadata
W20260929 22:50:40.065728 21872 connection.cc:535] server connection from 127.0.0.1:53482 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260929 22:50:40.066059 22011 connection.cc:441] server connection from 127.0.0.1:41222 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:41222 (request call id 1) could send its response
I20260929 22:50:40.074348 21992 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:50:40.067626 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60080 (local address 127.14.147.252:41485)
0929 22:50:40.067669 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0929 22:50:40.067671 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 22:50:40.067679 (+ 8us) server_negotiation.cc:407] Connection header received
0929 22:50:40.067758 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 22:50:40.067760 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 22:50:40.067773 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 22:50:40.067848 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0929 22:50:40.068497 (+ 649us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:50:40.069000 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:50:40.069683 (+ 683us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:50:40.069882 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:50:40.069939 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 22:50:40.071497 (+ 1558us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 22:50:40.071499 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 22:50:40.071499 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 22:50:40.071508 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0929 22:50:40.073077 (+ 1569us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 22:50:40.073455 (+ 378us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 22:50:40.073456 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 22:50:40.073456 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 22:50:40.073518 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 22:50:40.073705 (+ 187us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 22:50:40.073706 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 22:50:40.073707 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 22:50:40.073851 (+ 144us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 22:50:40.073928 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0929 22:50:40.074158 (+ 230us) server_negotiation.cc:299] Negotiation successful
0929 22:50:40.074250 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260929 22:50:40.076269 22274 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:50:40.067499 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56830 (local address 127.14.147.253:40121)
0929 22:50:40.067549 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0929 22:50:40.067551 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 22:50:40.067934 (+ 383us) server_negotiation.cc:407] Connection header received
0929 22:50:40.067992 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 22:50:40.067993 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 22:50:40.068009 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 22:50:40.068057 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0929 22:50:40.068824 (+ 767us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:50:40.069588 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:50:40.070707 (+ 1119us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:50:40.071033 (+ 326us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:50:40.071115 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 22:50:40.073026 (+ 1911us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 22:50:40.073028 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 22:50:40.073028 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 22:50:40.073037 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0929 22:50:40.074749 (+ 1712us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 22:50:40.075353 (+ 604us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 22:50:40.075355 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 22:50:40.075355 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 22:50:40.075451 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 22:50:40.075663 (+ 212us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 22:50:40.075664 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 22:50:40.075664 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 22:50:40.075818 (+ 154us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 22:50:40.075927 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0929 22:50:40.076067 (+ 140us) server_negotiation.cc:299] Negotiation successful
0929 22:50:40.076165 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260929 22:50:40.078779 22272 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:50:40.059853 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60070 (local address 127.14.147.252:41485)
0929 22:50:40.059895 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0929 22:50:40.059898 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 22:50:40.071614 (+ 11716us) server_negotiation.cc:407] Connection header received
0929 22:50:40.071661 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 22:50:40.071662 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 22:50:40.071674 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 22:50:40.071723 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0929 22:50:40.072474 (+ 751us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:50:40.073252 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:50:40.074314 (+ 1062us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:50:40.074598 (+ 284us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:50:40.074640 (+ 42us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 22:50:40.076005 (+ 1365us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 22:50:40.076007 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 22:50:40.076007 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 22:50:40.076014 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0929 22:50:40.077698 (+ 1684us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 22:50:40.078005 (+ 307us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 22:50:40.078006 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 22:50:40.078006 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 22:50:40.078057 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 22:50:40.078253 (+ 196us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 22:50:40.078254 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 22:50:40.078254 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 22:50:40.078401 (+ 147us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 22:50:40.078507 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0929 22:50:40.078593 (+ 86us) server_negotiation.cc:299] Negotiation successful
0929 22:50:40.078682 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260929 22:50:41.052083 22322 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:50:41.051812 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.147.254:45601 (local address 127.14.147.195:57849)
0929 22:50:41.051946 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0929 22:50:41.052031 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.147.254:45601: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":60,"threads_started":1}
W20260929 22:50:41.052371 22260 heartbeater.cc:644] Failed to heartbeat to 127.14.147.254:45601 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.14.147.254:45601: connect: Connection refused (error 111)
I20260929 22:50:41.053050 22323 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:50:41.052773 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.147.254:45601 (local address 127.14.147.194:47141)
0929 22:50:41.052942 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0929 22:50:41.053006 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.147.254:45601: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":155,"thread_start_us":51,"threads_started":1}
W20260929 22:50:41.053221 22123 heartbeater.cc:644] Failed to heartbeat to 127.14.147.254:45601 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.14.147.254:45601: connect: Connection refused (error 111)
I20260929 22:50:41.054021 22324 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:50:41.053811 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.147.254:45601 (local address 127.14.147.193:34723)
0929 22:50:41.053928 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0929 22:50:41.053986 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.147.254:45601: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":101,"thread_start_us":55,"threads_started":1}
W20260929 22:50:41.054195 21986 heartbeater.cc:644] Failed to heartbeat to 127.14.147.254:45601 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.14.147.254:45601: connect: Connection refused (error 111)
I20260929 22:50:41.558117 22325 raft_consensus.cc:482] T 00000000000000000000000000000000 P 98a8b44ee91645f8a3dc6f02d1c5c698 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d38d0f6f0d3c4d2b8b7131e3f0f72ffe)
I20260929 22:50:41.558117 22326 raft_consensus.cc:482] T 00000000000000000000000000000000 P e820b147283e4503b216af02b09dce4a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d38d0f6f0d3c4d2b8b7131e3f0f72ffe)
I20260929 22:50:41.558243 22325 raft_consensus.cc:504] T 00000000000000000000000000000000 P 98a8b44ee91645f8a3dc6f02d1c5c698 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 12 OBSOLETE_local: true peers { permanent_uuid: "d38d0f6f0d3c4d2b8b7131e3f0f72ffe" member_type: VOTER last_known_addr { host: "127.14.147.254" port: 45601 } } peers { permanent_uuid: "e820b147283e4503b216af02b09dce4a" member_type: VOTER last_known_addr { host: "127.14.147.253" port: 40121 } attrs { promote: false } } peers { permanent_uuid: "98a8b44ee91645f8a3dc6f02d1c5c698" member_type: VOTER last_known_addr { host: "127.14.147.252" port: 41485 } attrs { promote: false } }
I20260929 22:50:41.558259 22326 raft_consensus.cc:504] T 00000000000000000000000000000000 P e820b147283e4503b216af02b09dce4a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 12 OBSOLETE_local: true peers { permanent_uuid: "d38d0f6f0d3c4d2b8b7131e3f0f72ffe" member_type: VOTER last_known_addr { host: "127.14.147.254" port: 45601 } } peers { permanent_uuid: "e820b147283e4503b216af02b09dce4a" member_type: VOTER last_known_addr { host: "127.14.147.253" port: 40121 } attrs { promote: false } } peers { permanent_uuid: "98a8b44ee91645f8a3dc6f02d1c5c698" member_type: VOTER last_known_addr { host: "127.14.147.252" port: 41485 } attrs { promote: false } }
I20260929 22:50:41.558514 22326 leader_election.cc:288] T 00000000000000000000000000000000 P e820b147283e4503b216af02b09dce4a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d38d0f6f0d3c4d2b8b7131e3f0f72ffe (127.14.147.254:45601), 98a8b44ee91645f8a3dc6f02d1c5c698 (127.14.147.252:41485)
I20260929 22:50:41.558517 22325 leader_election.cc:288] T 00000000000000000000000000000000 P 98a8b44ee91645f8a3dc6f02d1c5c698 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d38d0f6f0d3c4d2b8b7131e3f0f72ffe (127.14.147.254:45601), e820b147283e4503b216af02b09dce4a (127.14.147.253:40121)
I20260929 22:50:41.558866 22327 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:50:41.558620 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.147.254:45601 (local address 127.0.0.1:53634)
0929 22:50:41.558755 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0929 22:50:41.558821 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.147.254:45601: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":50,"threads_started":1}
W20260929 22:50:41.559005 21604 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.14.147.254:45601: connect: Connection refused (error 111)
I20260929 22:50:41.559088 22328 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:50:41.558669 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.147.254:45601 (local address 127.0.0.1:53644)
0929 22:50:41.559003 (+ 334us) negotiation.cc:107] Waiting for socket to connect
0929 22:50:41.559049 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.147.254:45601: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":323,"thread_start_us":39,"threads_started":1}
W20260929 22:50:41.559175 21792 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.14.147.254:45601: connect: Connection refused (error 111)
I20260929 22:50:41.559573 22328 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:50:41.559492 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.147.254:45601 (local address 127.0.0.1:53654)
0929 22:50:41.559527 (+ 35us) negotiation.cc:107] Waiting for socket to connect
0929 22:50:41.559549 (+ 22us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.147.254:45601: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":22}
W20260929 22:50:41.559643 21792 leader_election.cc:334] T 00000000000000000000000000000000 P 98a8b44ee91645f8a3dc6f02d1c5c698 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer d38d0f6f0d3c4d2b8b7131e3f0f72ffe (127.14.147.254:45601): Network error: Client connection negotiation failed: client connection to 127.14.147.254:45601: connect: Connection refused (error 111)
I20260929 22:50:41.560108 22327 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:50:41.559536 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.147.254:45601 (local address 127.0.0.1:53666)
0929 22:50:41.560051 (+ 515us) negotiation.cc:107] Waiting for socket to connect
0929 22:50:41.560080 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.147.254:45601: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":499}
W20260929 22:50:41.560199 21604 leader_election.cc:334] T 00000000000000000000000000000000 P e820b147283e4503b216af02b09dce4a [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer d38d0f6f0d3c4d2b8b7131e3f0f72ffe (127.14.147.254:45601): Network error: Client connection negotiation failed: client connection to 127.14.147.254:45601: connect: Connection refused (error 111)
I20260929 22:50:41.561635 22332 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:50:41.558876 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60086 (local address 127.14.147.252:41485)
0929 22:50:41.558961 (+ 85us) server_negotiation.cc:206] Beginning negotiation
0929 22:50:41.558963 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 22:50:41.558970 (+ 7us) server_negotiation.cc:407] Connection header received
0929 22:50:41.558996 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 22:50:41.558997 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 22:50:41.559007 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 22:50:41.559246 (+ 239us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 22:50:41.559284 (+ 38us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:50:41.559947 (+ 663us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:50:41.560967 (+ 1020us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:50:41.561332 (+ 365us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:50:41.561393 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 22:50:41.561403 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0929 22:50:41.561528 (+ 125us) server_negotiation.cc:299] Negotiation successful
0929 22:50:41.561571 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68,"thread_start_us":36,"threads_started":1}
I20260929 22:50:41.561683 22330 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:50:41.558678 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.147.252:41485 (local address 127.0.0.1:60086)
0929 22:50:41.558770 (+ 92us) negotiation.cc:107] Waiting for socket to connect
0929 22:50:41.558780 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0929 22:50:41.558835 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 22:50:41.559113 (+ 278us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 22:50:41.559114 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 22:50:41.559117 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 22:50:41.559182 (+ 65us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 22:50:41.559184 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 22:50:41.560027 (+ 843us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 22:50:41.560028 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 22:50:41.560887 (+ 859us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 22:50:41.560889 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 22:50:41.561401 (+ 512us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 22:50:41.561402 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 22:50:41.561471 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 22:50:41.561474 (+ 3us) client_negotiation.cc:769] Sending connection context
0929 22:50:41.561517 (+ 43us) client_negotiation.cc:240] Negotiation successful
0929 22:50:41.561571 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":80,"thread_start_us":37,"threads_started":1}
I20260929 22:50:41.561731 22329 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:50:41.558663 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.147.253:40121 (local address 127.0.0.1:56840)
0929 22:50:41.558967 (+ 304us) negotiation.cc:107] Waiting for socket to connect
0929 22:50:41.558976 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0929 22:50:41.559042 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 22:50:41.559225 (+ 183us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 22:50:41.559226 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 22:50:41.559228 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 22:50:41.559277 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 22:50:41.559279 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 22:50:41.560053 (+ 774us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 22:50:41.560054 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 22:50:41.560948 (+ 894us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 22:50:41.560951 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 22:50:41.561457 (+ 506us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 22:50:41.561458 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 22:50:41.561521 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 22:50:41.561524 (+ 3us) client_negotiation.cc:769] Sending connection context
0929 22:50:41.561570 (+ 46us) client_negotiation.cc:240] Negotiation successful
0929 22:50:41.561616 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":290,"thread_start_us":54,"threads_started":1}
I20260929 22:50:41.561872 22331 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:50:41.558801 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56840 (local address 127.14.147.253:40121)
0929 22:50:41.558924 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0929 22:50:41.558926 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 22:50:41.559019 (+ 93us) server_negotiation.cc:407] Connection header received
0929 22:50:41.559107 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 22:50:41.559108 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 22:50:41.559117 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 22:50:41.559159 (+ 42us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 22:50:41.559368 (+ 209us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:50:41.559980 (+ 612us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:50:41.561032 (+ 1052us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:50:41.561391 (+ 359us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:50:41.561451 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 22:50:41.561459 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0929 22:50:41.561578 (+ 119us) server_negotiation.cc:299] Negotiation successful
0929 22:50:41.561619 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":109,"thread_start_us":46,"threads_started":1}
I20260929 22:50:41.561909 21815 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "e820b147283e4503b216af02b09dce4a" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "98a8b44ee91645f8a3dc6f02d1c5c698" is_pre_election: true
I20260929 22:50:41.562022 21815 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 98a8b44ee91645f8a3dc6f02d1c5c698 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e820b147283e4503b216af02b09dce4a in term 1.
I20260929 22:50:41.562026 21627 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "98a8b44ee91645f8a3dc6f02d1c5c698" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "e820b147283e4503b216af02b09dce4a" is_pre_election: true
I20260929 22:50:41.562111 21627 raft_consensus.cc:2454] T 00000000000000000000000000000000 P e820b147283e4503b216af02b09dce4a [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 98a8b44ee91645f8a3dc6f02d1c5c698 in term 1.
I20260929 22:50:41.562170 21603 leader_election.cc:302] T 00000000000000000000000000000000 P e820b147283e4503b216af02b09dce4a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 98a8b44ee91645f8a3dc6f02d1c5c698, e820b147283e4503b216af02b09dce4a; no voters: d38d0f6f0d3c4d2b8b7131e3f0f72ffe
I20260929 22:50:41.562253 21789 leader_election.cc:302] T 00000000000000000000000000000000 P 98a8b44ee91645f8a3dc6f02d1c5c698 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 98a8b44ee91645f8a3dc6f02d1c5c698, e820b147283e4503b216af02b09dce4a; no voters: d38d0f6f0d3c4d2b8b7131e3f0f72ffe
I20260929 22:50:41.562297 22326 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e820b147283e4503b216af02b09dce4a [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260929 22:50:41.562333 22326 raft_consensus.cc:482] T 00000000000000000000000000000000 P e820b147283e4503b216af02b09dce4a [term 1 FOLLOWER]: Starting leader election (detected failure of leader d38d0f6f0d3c4d2b8b7131e3f0f72ffe)
I20260929 22:50:41.562351 22326 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e820b147283e4503b216af02b09dce4a [term 1 FOLLOWER]: Advancing to term 2
I20260929 22:50:41.562368 22325 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 98a8b44ee91645f8a3dc6f02d1c5c698 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260929 22:50:41.562402 22325 raft_consensus.cc:482] T 00000000000000000000000000000000 P 98a8b44ee91645f8a3dc6f02d1c5c698 [term 1 FOLLOWER]: Starting leader election (detected failure of leader d38d0f6f0d3c4d2b8b7131e3f0f72ffe)
I20260929 22:50:41.562419 22325 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 98a8b44ee91645f8a3dc6f02d1c5c698 [term 1 FOLLOWER]: Advancing to term 2
I20260929 22:50:41.563200 22325 raft_consensus.cc:504] T 00000000000000000000000000000000 P 98a8b44ee91645f8a3dc6f02d1c5c698 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 12 OBSOLETE_local: true peers { permanent_uuid: "d38d0f6f0d3c4d2b8b7131e3f0f72ffe" member_type: VOTER last_known_addr { host: "127.14.147.254" port: 45601 } } peers { permanent_uuid: "e820b147283e4503b216af02b09dce4a" member_type: VOTER last_known_addr { host: "127.14.147.253" port: 40121 } attrs { promote: false } } peers { permanent_uuid: "98a8b44ee91645f8a3dc6f02d1c5c698" member_type: VOTER last_known_addr { host: "127.14.147.252" port: 41485 } attrs { promote: false } }
I20260929 22:50:41.563200 22326 raft_consensus.cc:504] T 00000000000000000000000000000000 P e820b147283e4503b216af02b09dce4a [term 2 FOLLOWER]: Starting leader election with config: opid_index: 12 OBSOLETE_local: true peers { permanent_uuid: "d38d0f6f0d3c4d2b8b7131e3f0f72ffe" member_type: VOTER last_known_addr { host: "127.14.147.254" port: 45601 } } peers { permanent_uuid: "e820b147283e4503b216af02b09dce4a" member_type: VOTER last_known_addr { host: "127.14.147.253" port: 40121 } attrs { promote: false } } peers { permanent_uuid: "98a8b44ee91645f8a3dc6f02d1c5c698" member_type: VOTER last_known_addr { host: "127.14.147.252" port: 41485 } attrs { promote: false } }
I20260929 22:50:41.563333 22326 leader_election.cc:288] T 00000000000000000000000000000000 P e820b147283e4503b216af02b09dce4a [CANDIDATE]: Term 2 election: Requested vote from peers d38d0f6f0d3c4d2b8b7131e3f0f72ffe (127.14.147.254:45601), 98a8b44ee91645f8a3dc6f02d1c5c698 (127.14.147.252:41485)
I20260929 22:50:41.563333 22325 leader_election.cc:288] T 00000000000000000000000000000000 P 98a8b44ee91645f8a3dc6f02d1c5c698 [CANDIDATE]: Term 2 election: Requested vote from peers d38d0f6f0d3c4d2b8b7131e3f0f72ffe (127.14.147.254:45601), e820b147283e4503b216af02b09dce4a (127.14.147.253:40121)
I20260929 22:50:41.563541 21815 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "e820b147283e4503b216af02b09dce4a" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "98a8b44ee91645f8a3dc6f02d1c5c698"
I20260929 22:50:41.563598 22329 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:50:41.563494 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.147.254:45601 (local address 127.0.0.1:53668)
0929 22:50:41.563530 (+ 36us) negotiation.cc:107] Waiting for socket to connect
0929 22:50:41.563556 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.147.254:45601: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":21}
I20260929 22:50:41.563650 21815 raft_consensus.cc:2379] T 00000000000000000000000000000000 P 98a8b44ee91645f8a3dc6f02d1c5c698 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate e820b147283e4503b216af02b09dce4a in current term 2: Already voted for candidate 98a8b44ee91645f8a3dc6f02d1c5c698 in this term.
I20260929 22:50:41.563849 21627 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "98a8b44ee91645f8a3dc6f02d1c5c698" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "e820b147283e4503b216af02b09dce4a"
I20260929 22:50:41.563858 22330 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:50:41.563481 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.147.254:45601 (local address 127.0.0.1:53670)
0929 22:50:41.563803 (+ 322us) negotiation.cc:107] Waiting for socket to connect
0929 22:50:41.563831 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.147.254:45601: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":306}
I20260929 22:50:41.563948 21627 raft_consensus.cc:2379] T 00000000000000000000000000000000 P e820b147283e4503b216af02b09dce4a [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 98a8b44ee91645f8a3dc6f02d1c5c698 in current term 2: Already voted for candidate e820b147283e4503b216af02b09dce4a in this term.
I20260929 22:50:41.564070 22329 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:50:41.563803 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.147.254:45601 (local address 127.0.0.1:53684)
0929 22:50:41.564021 (+ 218us) negotiation.cc:107] Waiting for socket to connect
0929 22:50:41.564049 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.147.254:45601: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":202}
W20260929 22:50:41.564131 21792 leader_election.cc:334] T 00000000000000000000000000000000 P 98a8b44ee91645f8a3dc6f02d1c5c698 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer d38d0f6f0d3c4d2b8b7131e3f0f72ffe (127.14.147.254:45601): Network error: Client connection negotiation failed: client connection to 127.14.147.254:45601: connect: Connection refused (error 111)
I20260929 22:50:41.564193 22330 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:50:41.564113 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.147.254:45601 (local address 127.0.0.1:53690)
0929 22:50:41.564146 (+ 33us) negotiation.cc:107] Waiting for socket to connect
0929 22:50:41.564164 (+ 18us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.147.254:45601: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":22}
I20260929 22:50:41.564199 21792 leader_election.cc:302] T 00000000000000000000000000000000 P 98a8b44ee91645f8a3dc6f02d1c5c698 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 98a8b44ee91645f8a3dc6f02d1c5c698; no voters: d38d0f6f0d3c4d2b8b7131e3f0f72ffe, e820b147283e4503b216af02b09dce4a
W20260929 22:50:41.564241 21604 leader_election.cc:334] T 00000000000000000000000000000000 P e820b147283e4503b216af02b09dce4a [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer d38d0f6f0d3c4d2b8b7131e3f0f72ffe (127.14.147.254:45601): Network error: Client connection negotiation failed: client connection to 127.14.147.254:45601: connect: Connection refused (error 111)
I20260929 22:50:41.564270 21604 leader_election.cc:302] T 00000000000000000000000000000000 P e820b147283e4503b216af02b09dce4a [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e820b147283e4503b216af02b09dce4a; no voters: 98a8b44ee91645f8a3dc6f02d1c5c698, d38d0f6f0d3c4d2b8b7131e3f0f72ffe
I20260929 22:50:41.564296 22325 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 98a8b44ee91645f8a3dc6f02d1c5c698 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260929 22:50:41.564327 22326 raft_consensus.cc:2726] T 00000000000000000000000000000000 P e820b147283e4503b216af02b09dce4a [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260929 22:50:42.052984 22335 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:50:42.052698 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.147.254:45601 (local address 127.14.147.195:44305)
0929 22:50:42.052857 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0929 22:50:42.052921 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.147.254:45601: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":66,"threads_started":1}
I20260929 22:50:42.053828 22336 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:50:42.053574 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.147.254:45601 (local address 127.14.147.194:60279)
0929 22:50:42.053755 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0929 22:50:42.053790 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.147.254:45601: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":165,"thread_start_us":70,"threads_started":1}
I20260929 22:50:42.054669 22337 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:50:42.054463 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.147.254:45601 (local address 127.14.147.193:42153)
0929 22:50:42.054576 (+ 113us) negotiation.cc:107] Waiting for socket to connect
0929 22:50:42.054613 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.147.254:45601: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":96,"thread_start_us":52,"threads_started":1}
I20260929 22:50:43.053622 22338 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:50:43.053423 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.147.254:45601 (local address 127.14.147.195:35463)
0929 22:50:43.053525 (+ 102us) negotiation.cc:107] Waiting for socket to connect
0929 22:50:43.053564 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.147.254:45601: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":83,"thread_start_us":61,"threads_started":1}
I20260929 22:50:43.054404 22339 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:50:43.054202 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.147.254:45601 (local address 127.14.147.194:34915)
0929 22:50:43.054323 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0929 22:50:43.054358 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.147.254:45601: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":105,"thread_start_us":78,"threads_started":1}
I20260929 22:50:43.055209 22340 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:50:43.055059 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.147.254:45601 (local address 127.14.147.193:36119)
0929 22:50:43.055150 (+ 91us) negotiation.cc:107] Waiting for socket to connect
0929 22:50:43.055175 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.147.254:45601: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":78,"thread_start_us":44,"threads_started":1}
I20260929 22:50:43.389191 22341 raft_consensus.cc:482] T 00000000000000000000000000000000 P 98a8b44ee91645f8a3dc6f02d1c5c698 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 22:50:43.389302 22341 raft_consensus.cc:504] T 00000000000000000000000000000000 P 98a8b44ee91645f8a3dc6f02d1c5c698 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 12 OBSOLETE_local: true peers { permanent_uuid: "d38d0f6f0d3c4d2b8b7131e3f0f72ffe" member_type: VOTER last_known_addr { host: "127.14.147.254" port: 45601 } } peers { permanent_uuid: "e820b147283e4503b216af02b09dce4a" member_type: VOTER last_known_addr { host: "127.14.147.253" port: 40121 } attrs { promote: false } } peers { permanent_uuid: "98a8b44ee91645f8a3dc6f02d1c5c698" member_type: VOTER last_known_addr { host: "127.14.147.252" port: 41485 } attrs { promote: false } }
I20260929 22:50:43.389490 22341 leader_election.cc:288] T 00000000000000000000000000000000 P 98a8b44ee91645f8a3dc6f02d1c5c698 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers d38d0f6f0d3c4d2b8b7131e3f0f72ffe (127.14.147.254:45601), e820b147283e4503b216af02b09dce4a (127.14.147.253:40121)
I20260929 22:50:43.389827 22342 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:50:43.389642 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.147.254:45601 (local address 127.0.0.1:53910)
0929 22:50:43.389749 (+ 107us) negotiation.cc:107] Waiting for socket to connect
0929 22:50:43.389789 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.147.254:45601: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":90,"thread_start_us":54,"threads_started":1}
I20260929 22:50:43.389935 21627 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "98a8b44ee91645f8a3dc6f02d1c5c698" candidate_term: 3 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "e820b147283e4503b216af02b09dce4a" is_pre_election: true
I20260929 22:50:43.390129 21627 raft_consensus.cc:2454] T 00000000000000000000000000000000 P e820b147283e4503b216af02b09dce4a [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 98a8b44ee91645f8a3dc6f02d1c5c698 in term 2.
I20260929 22:50:43.390352 22342 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:50:43.390266 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.147.254:45601 (local address 127.0.0.1:53926)
0929 22:50:43.390300 (+ 34us) negotiation.cc:107] Waiting for socket to connect
0929 22:50:43.390327 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.147.254:45601: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":18}
I20260929 22:50:43.390381 21789 leader_election.cc:302] T 00000000000000000000000000000000 P 98a8b44ee91645f8a3dc6f02d1c5c698 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 98a8b44ee91645f8a3dc6f02d1c5c698, e820b147283e4503b216af02b09dce4a; no voters:
I20260929 22:50:43.390471 22341 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 98a8b44ee91645f8a3dc6f02d1c5c698 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260929 22:50:43.390540 22341 raft_consensus.cc:482] T 00000000000000000000000000000000 P 98a8b44ee91645f8a3dc6f02d1c5c698 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260929 22:50:43.390473 21792 leader_election.cc:334] T 00000000000000000000000000000000 P 98a8b44ee91645f8a3dc6f02d1c5c698 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer d38d0f6f0d3c4d2b8b7131e3f0f72ffe (127.14.147.254:45601): Network error: Client connection negotiation failed: client connection to 127.14.147.254:45601: connect: Connection refused (error 111)
I20260929 22:50:43.390595 22341 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 98a8b44ee91645f8a3dc6f02d1c5c698 [term 2 FOLLOWER]: Advancing to term 3
I20260929 22:50:43.391359 22341 raft_consensus.cc:504] T 00000000000000000000000000000000 P 98a8b44ee91645f8a3dc6f02d1c5c698 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 12 OBSOLETE_local: true peers { permanent_uuid: "d38d0f6f0d3c4d2b8b7131e3f0f72ffe" member_type: VOTER last_known_addr { host: "127.14.147.254" port: 45601 } } peers { permanent_uuid: "e820b147283e4503b216af02b09dce4a" member_type: VOTER last_known_addr { host: "127.14.147.253" port: 40121 } attrs { promote: false } } peers { permanent_uuid: "98a8b44ee91645f8a3dc6f02d1c5c698" member_type: VOTER last_known_addr { host: "127.14.147.252" port: 41485 } attrs { promote: false } }
I20260929 22:50:43.391546 22341 leader_election.cc:288] T 00000000000000000000000000000000 P 98a8b44ee91645f8a3dc6f02d1c5c698 [CANDIDATE]: Term 3 election: Requested vote from peers d38d0f6f0d3c4d2b8b7131e3f0f72ffe (127.14.147.254:45601), e820b147283e4503b216af02b09dce4a (127.14.147.253:40121)
I20260929 22:50:43.391793 22342 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:50:43.391693 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.147.254:45601 (local address 127.0.0.1:53936)
0929 22:50:43.391738 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0929 22:50:43.391768 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.147.254:45601: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
I20260929 22:50:43.391794 21627 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "98a8b44ee91645f8a3dc6f02d1c5c698" candidate_term: 3 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "e820b147283e4503b216af02b09dce4a"
I20260929 22:50:43.391907 21627 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e820b147283e4503b216af02b09dce4a [term 2 FOLLOWER]: Advancing to term 3
I20260929 22:50:43.392123 22342 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:50:43.392039 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.147.254:45601 (local address 127.0.0.1:53946)
0929 22:50:43.392078 (+ 39us) negotiation.cc:107] Waiting for socket to connect
0929 22:50:43.392102 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.147.254:45601: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":26}
W20260929 22:50:43.392187 21792 leader_election.cc:334] T 00000000000000000000000000000000 P 98a8b44ee91645f8a3dc6f02d1c5c698 [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer d38d0f6f0d3c4d2b8b7131e3f0f72ffe (127.14.147.254:45601): Network error: Client connection negotiation failed: client connection to 127.14.147.254:45601: connect: Connection refused (error 111)
I20260929 22:50:43.392606 21627 raft_consensus.cc:2454] T 00000000000000000000000000000000 P e820b147283e4503b216af02b09dce4a [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 98a8b44ee91645f8a3dc6f02d1c5c698 in term 3.
I20260929 22:50:43.392799 21789 leader_election.cc:302] T 00000000000000000000000000000000 P 98a8b44ee91645f8a3dc6f02d1c5c698 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 98a8b44ee91645f8a3dc6f02d1c5c698, e820b147283e4503b216af02b09dce4a; no voters: d38d0f6f0d3c4d2b8b7131e3f0f72ffe
I20260929 22:50:43.392895 22341 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 98a8b44ee91645f8a3dc6f02d1c5c698 [term 3 FOLLOWER]: Leader election won for term 3
I20260929 22:50:43.393006 22341 raft_consensus.cc:686] T 00000000000000000000000000000000 P 98a8b44ee91645f8a3dc6f02d1c5c698 [term 3 LEADER]: Becoming Leader. State: Replica: 98a8b44ee91645f8a3dc6f02d1c5c698, State: Running, Role: LEADER
I20260929 22:50:43.393074 22341 consensus_queue.cc:237] T 00000000000000000000000000000000 P 98a8b44ee91645f8a3dc6f02d1c5c698 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 1.12, Last appended by leader: 12, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 12 OBSOLETE_local: true peers { permanent_uuid: "d38d0f6f0d3c4d2b8b7131e3f0f72ffe" member_type: VOTER last_known_addr { host: "127.14.147.254" port: 45601 } } peers { permanent_uuid: "e820b147283e4503b216af02b09dce4a" member_type: VOTER last_known_addr { host: "127.14.147.253" port: 40121 } attrs { promote: false } } peers { permanent_uuid: "98a8b44ee91645f8a3dc6f02d1c5c698" member_type: VOTER last_known_addr { host: "127.14.147.252" port: 41485 } attrs { promote: false } }
I20260929 22:50:43.393453 22344 sys_catalog.cc:455] T 00000000000000000000000000000000 P 98a8b44ee91645f8a3dc6f02d1c5c698 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 98a8b44ee91645f8a3dc6f02d1c5c698. Latest consensus state: current_term: 3 leader_uuid: "98a8b44ee91645f8a3dc6f02d1c5c698" committed_config { opid_index: 12 OBSOLETE_local: true peers { permanent_uuid: "d38d0f6f0d3c4d2b8b7131e3f0f72ffe" member_type: VOTER last_known_addr { host: "127.14.147.254" port: 45601 } } peers { permanent_uuid: "e820b147283e4503b216af02b09dce4a" member_type: VOTER last_known_addr { host: "127.14.147.253" port: 40121 } attrs { promote: false } } peers { permanent_uuid: "98a8b44ee91645f8a3dc6f02d1c5c698" member_type: VOTER last_known_addr { host: "127.14.147.252" port: 41485 } attrs { promote: false } } }
I20260929 22:50:43.393523 22344 sys_catalog.cc:458] T 00000000000000000000000000000000 P 98a8b44ee91645f8a3dc6f02d1c5c698 [sys.catalog]: This master's current role is: LEADER
I20260929 22:50:43.393679 22346 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 22:50:43.394068 22346 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 22:50:43.394129 22346 catalog_manager.cc:1259] Loaded cluster ID: eeb78789c1ae4d64a674c24edf4ad136
I20260929 22:50:43.394145 22346 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 22:50:43.394317 22346 catalog_manager.cc:1504] Loading token signing keys...
I20260929 22:50:43.394393 22346 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 98a8b44ee91645f8a3dc6f02d1c5c698: Loaded TSK: 0
I20260929 22:50:43.394538 22346 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 22:50:43.484339 21803 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60080:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 22:50:43.485939 21803 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 22:50:43.486999 22342 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:50:43.486880 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.147.254:45601 (local address 127.0.0.1:53956)
0929 22:50:43.486931 (+ 51us) negotiation.cc:107] Waiting for socket to connect
0929 22:50:43.486967 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.147.254:45601: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
I20260929 22:50:43.487025 21627 raft_consensus.cc:1261] T 00000000000000000000000000000000 P e820b147283e4503b216af02b09dce4a [term 3 FOLLOWER]: Refusing update from remote peer 98a8b44ee91645f8a3dc6f02d1c5c698: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 3 index: 14. (index mismatch)
I20260929 22:50:43.487244 22344 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 98a8b44ee91645f8a3dc6f02d1c5c698 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e820b147283e4503b216af02b09dce4a" member_type: VOTER last_known_addr { host: "127.14.147.253" port: 40121 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20260929 22:50:43.487385 22342 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:50:43.487288 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.147.254:45601 (local address 127.0.0.1:53960)
0929 22:50:43.487334 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0929 22:50:43.487363 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.147.254:45601: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
W20260929 22:50:43.487529 21792 consensus_peers.cc:487] T 00000000000000000000000000000000 P 98a8b44ee91645f8a3dc6f02d1c5c698 -> Peer d38d0f6f0d3c4d2b8b7131e3f0f72ffe (127.14.147.254:45601): Couldn't send request to peer d38d0f6f0d3c4d2b8b7131e3f0f72ffe. Status: Network error: Client connection negotiation failed: client connection to 127.14.147.254:45601: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260929 22:50:43.487862 22348 sys_catalog.cc:455] T 00000000000000000000000000000000 P e820b147283e4503b216af02b09dce4a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 98a8b44ee91645f8a3dc6f02d1c5c698. Latest consensus state: current_term: 3 leader_uuid: "98a8b44ee91645f8a3dc6f02d1c5c698" committed_config { opid_index: 12 OBSOLETE_local: true peers { permanent_uuid: "d38d0f6f0d3c4d2b8b7131e3f0f72ffe" member_type: VOTER last_known_addr { host: "127.14.147.254" port: 45601 } } peers { permanent_uuid: "e820b147283e4503b216af02b09dce4a" member_type: VOTER last_known_addr { host: "127.14.147.253" port: 40121 } attrs { promote: false } } peers { permanent_uuid: "98a8b44ee91645f8a3dc6f02d1c5c698" member_type: VOTER last_known_addr { host: "127.14.147.252" port: 41485 } attrs { promote: false } } }
I20260929 22:50:43.487955 22348 sys_catalog.cc:458] T 00000000000000000000000000000000 P e820b147283e4503b216af02b09dce4a [sys.catalog]: This master's current role is: FOLLOWER
I20260929 22:50:43.488387 22344 sys_catalog.cc:455] T 00000000000000000000000000000000 P 98a8b44ee91645f8a3dc6f02d1c5c698 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 3 leader_uuid: "98a8b44ee91645f8a3dc6f02d1c5c698" committed_config { opid_index: 12 OBSOLETE_local: true peers { permanent_uuid: "d38d0f6f0d3c4d2b8b7131e3f0f72ffe" member_type: VOTER last_known_addr { host: "127.14.147.254" port: 45601 } } peers { permanent_uuid: "e820b147283e4503b216af02b09dce4a" member_type: VOTER last_known_addr { host: "127.14.147.253" port: 40121 } attrs { promote: false } } peers { permanent_uuid: "98a8b44ee91645f8a3dc6f02d1c5c698" member_type: VOTER last_known_addr { host: "127.14.147.252" port: 41485 } attrs { promote: false } } }
I20260929 22:50:43.488484 22344 sys_catalog.cc:458] T 00000000000000000000000000000000 P 98a8b44ee91645f8a3dc6f02d1c5c698 [sys.catalog]: This master's current role is: LEADER
I20260929 22:50:43.488521 22349 mvcc.cc:205] Tried to move back new op lower bound from 7334798309320646656 to 7334798308938616832. Current Snapshot: MvccSnapshot[applied={T|T < 7334798309320646656}]
I20260929 22:50:43.488675 22348 sys_catalog.cc:455] T 00000000000000000000000000000000 P e820b147283e4503b216af02b09dce4a [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 3 leader_uuid: "98a8b44ee91645f8a3dc6f02d1c5c698" committed_config { opid_index: 12 OBSOLETE_local: true peers { permanent_uuid: "d38d0f6f0d3c4d2b8b7131e3f0f72ffe" member_type: VOTER last_known_addr { host: "127.14.147.254" port: 45601 } } peers { permanent_uuid: "e820b147283e4503b216af02b09dce4a" member_type: VOTER last_known_addr { host: "127.14.147.253" port: 40121 } attrs { promote: false } } peers { permanent_uuid: "98a8b44ee91645f8a3dc6f02d1c5c698" member_type: VOTER last_known_addr { host: "127.14.147.252" port: 41485 } attrs { promote: false } } }
I20260929 22:50:43.488754 22348 sys_catalog.cc:458] T 00000000000000000000000000000000 P e820b147283e4503b216af02b09dce4a [sys.catalog]: This master's current role is: FOLLOWER
I20260929 22:50:43.493434 22342 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:50:43.490136 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.147.195:38905 (local address 127.0.0.1:57304)
0929 22:50:43.490192 (+ 56us) negotiation.cc:107] Waiting for socket to connect
0929 22:50:43.490203 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0929 22:50:43.490255 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 22:50:43.490538 (+ 283us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 22:50:43.490539 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 22:50:43.490541 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 22:50:43.490601 (+ 60us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 22:50:43.490603 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 22:50:43.491303 (+ 700us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 22:50:43.491304 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 22:50:43.492073 (+ 769us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 22:50:43.492075 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 22:50:43.493179 (+ 1104us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 22:50:43.493180 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 22:50:43.493248 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 22:50:43.493253 (+ 5us) client_negotiation.cc:769] Sending connection context
0929 22:50:43.493302 (+ 49us) client_negotiation.cc:240] Negotiation successful
0929 22:50:43.493353 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":39}
I20260929 22:50:43.493711 22356 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:50:43.490310 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57304 (local address 127.14.147.195:38905)
0929 22:50:43.490421 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0929 22:50:43.490422 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0929 22:50:43.490430 (+ 8us) server_negotiation.cc:407] Connection header received
0929 22:50:43.490456 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 22:50:43.490457 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 22:50:43.490467 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 22:50:43.490600 (+ 133us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 22:50:43.490644 (+ 44us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:50:43.491239 (+ 595us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:50:43.492156 (+ 917us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:50:43.492882 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:50:43.493181 (+ 299us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 22:50:43.493215 (+ 34us) server_negotiation.cc:1035] Waiting for connection context
0929 22:50:43.493544 (+ 329us) server_negotiation.cc:299] Negotiation successful
0929 22:50:43.493626 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":91,"thread_start_us":64,"threads_started":1}
I20260929 22:50:43.493897 22355 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:50:43.490188 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.147.194:45475 (local address 127.0.0.1:41228)
0929 22:50:43.490370 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0929 22:50:43.490379 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0929 22:50:43.490426 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 22:50:43.490821 (+ 395us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 22:50:43.490822 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 22:50:43.490824 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 22:50:43.490855 (+ 31us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 22:50:43.490856 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 22:50:43.491814 (+ 958us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 22:50:43.491814 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 22:50:43.493000 (+ 1186us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 22:50:43.493002 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 22:50:43.493676 (+ 674us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 22:50:43.493677 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 22:50:43.493747 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 22:50:43.493751 (+ 4us) client_negotiation.cc:769] Sending connection context
0929 22:50:43.493778 (+ 27us) client_negotiation.cc:240] Negotiation successful
0929 22:50:43.493818 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":165,"mutex_wait_us":29,"thread_start_us":66,"threads_started":1}
I20260929 22:50:43.493912 22193 tablet_service.cc:1467] Processing CreateTablet for tablet a8b5ef3087834d89941bf92a2c409563 (DEFAULT_TABLE table=test-table [id=5a7b0f1d956a481ab858a187ce88370d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 22:50:43.494056 22193 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a8b5ef3087834d89941bf92a2c409563. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 22:50:43.494206 22358 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:50:43.490423 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41228 (local address 127.14.147.194:45475)
0929 22:50:43.490688 (+ 265us) server_negotiation.cc:206] Beginning negotiation
0929 22:50:43.490690 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 22:50:43.490697 (+ 7us) server_negotiation.cc:407] Connection header received
0929 22:50:43.490720 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 22:50:43.490721 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 22:50:43.490742 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 22:50:43.490808 (+ 66us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 22:50:43.490911 (+ 103us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:50:43.491706 (+ 795us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:50:43.493082 (+ 1376us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:50:43.493588 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:50:43.494069 (+ 481us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 22:50:43.494080 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0929 22:50:43.494098 (+ 18us) server_negotiation.cc:299] Negotiation successful
0929 22:50:43.494137 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":242,"thread_start_us":54,"threads_started":1}
I20260929 22:50:43.494231 22357 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:50:43.490339 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53494 (local address 127.14.147.193:44967)
0929 22:50:43.490511 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0929 22:50:43.490512 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0929 22:50:43.490520 (+ 8us) server_negotiation.cc:407] Connection header received
0929 22:50:43.490543 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 22:50:43.490544 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 22:50:43.490553 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 22:50:43.490612 (+ 59us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 22:50:43.490987 (+ 375us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:50:43.491738 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:50:43.493309 (+ 1571us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:50:43.493841 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:50:43.493908 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 22:50:43.493916 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0929 22:50:43.494100 (+ 184us) server_negotiation.cc:299] Negotiation successful
0929 22:50:43.494144 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":151,"thread_start_us":54,"threads_started":1}
I20260929 22:50:43.494400 22354 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:50:43.490259 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.147.193:44967 (local address 127.0.0.1:53494)
0929 22:50:43.490438 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0929 22:50:43.490446 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0929 22:50:43.490485 (+ 39us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 22:50:43.490855 (+ 370us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 22:50:43.490856 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 22:50:43.490859 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 22:50:43.490907 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 22:50:43.490909 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 22:50:43.491818 (+ 909us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 22:50:43.491819 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 22:50:43.493246 (+ 1427us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 22:50:43.493248 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 22:50:43.493947 (+ 699us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 22:50:43.493948 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 22:50:43.494013 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 22:50:43.494018 (+ 5us) client_negotiation.cc:769] Sending connection context
0929 22:50:43.494300 (+ 282us) client_negotiation.cc:240] Negotiation successful
0929 22:50:43.494343 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":163,"mutex_wait_us":45,"thread_start_us":64,"threads_started":1}
I20260929 22:50:43.494433 22056 tablet_service.cc:1467] Processing CreateTablet for tablet a8b5ef3087834d89941bf92a2c409563 (DEFAULT_TABLE table=test-table [id=5a7b0f1d956a481ab858a187ce88370d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 22:50:43.494553 22056 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a8b5ef3087834d89941bf92a2c409563. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 22:50:43.495376 22359 tablet_bootstrap.cc:492] T a8b5ef3087834d89941bf92a2c409563 P f9ed52b3ef574d3d8c17014dbf842a54: Bootstrap starting.
I20260929 22:50:43.495832 22360 tablet_bootstrap.cc:492] T a8b5ef3087834d89941bf92a2c409563 P ab4a15db2c3f422e8646d4b5255cf8df: Bootstrap starting.
I20260929 22:50:43.495849 22359 tablet_bootstrap.cc:654] T a8b5ef3087834d89941bf92a2c409563 P f9ed52b3ef574d3d8c17014dbf842a54: Neither blocks nor log segments found. Creating new log.
I20260929 22:50:43.496098 21919 tablet_service.cc:1467] Processing CreateTablet for tablet a8b5ef3087834d89941bf92a2c409563 (DEFAULT_TABLE table=test-table [id=5a7b0f1d956a481ab858a187ce88370d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 22:50:43.496176 22360 tablet_bootstrap.cc:654] T a8b5ef3087834d89941bf92a2c409563 P ab4a15db2c3f422e8646d4b5255cf8df: Neither blocks nor log segments found. Creating new log.
I20260929 22:50:43.496217 21919 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a8b5ef3087834d89941bf92a2c409563. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 22:50:43.496632 22360 tablet_bootstrap.cc:492] T a8b5ef3087834d89941bf92a2c409563 P ab4a15db2c3f422e8646d4b5255cf8df: No bootstrap required, opened a new log
I20260929 22:50:43.496670 22360 ts_tablet_manager.cc:1397] T a8b5ef3087834d89941bf92a2c409563 P ab4a15db2c3f422e8646d4b5255cf8df: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260929 22:50:43.496753 22360 raft_consensus.cc:348] T a8b5ef3087834d89941bf92a2c409563 P ab4a15db2c3f422e8646d4b5255cf8df [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9ed52b3ef574d3d8c17014dbf842a54" member_type: VOTER last_known_addr { host: "127.14.147.195" port: 38905 } } peers { permanent_uuid: "ab4a15db2c3f422e8646d4b5255cf8df" member_type: VOTER last_known_addr { host: "127.14.147.194" port: 45475 } } peers { permanent_uuid: "e0d74c61684f40338bc384332f90e9c7" member_type: VOTER last_known_addr { host: "127.14.147.193" port: 44967 } }
I20260929 22:50:43.496805 22360 raft_consensus.cc:374] T a8b5ef3087834d89941bf92a2c409563 P ab4a15db2c3f422e8646d4b5255cf8df [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 22:50:43.496819 22360 raft_consensus.cc:729] T a8b5ef3087834d89941bf92a2c409563 P ab4a15db2c3f422e8646d4b5255cf8df [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ab4a15db2c3f422e8646d4b5255cf8df, State: Initialized, Role: FOLLOWER
I20260929 22:50:43.496848 22360 consensus_queue.cc:260] T a8b5ef3087834d89941bf92a2c409563 P ab4a15db2c3f422e8646d4b5255cf8df [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9ed52b3ef574d3d8c17014dbf842a54" member_type: VOTER last_known_addr { host: "127.14.147.195" port: 38905 } } peers { permanent_uuid: "ab4a15db2c3f422e8646d4b5255cf8df" member_type: VOTER last_known_addr { host: "127.14.147.194" port: 45475 } } peers { permanent_uuid: "e0d74c61684f40338bc384332f90e9c7" member_type: VOTER last_known_addr { host: "127.14.147.193" port: 44967 } }
I20260929 22:50:43.497000 22360 ts_tablet_manager.cc:1428] T a8b5ef3087834d89941bf92a2c409563 P ab4a15db2c3f422e8646d4b5255cf8df: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260929 22:50:43.497300 22339 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:50:43.497199 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.147.254:45601 (local address 127.14.147.194:40087)
0929 22:50:43.497237 (+ 38us) negotiation.cc:107] Waiting for socket to connect
0929 22:50:43.497271 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.147.254:45601: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":21}
I20260929 22:50:43.497578 22364 tablet_bootstrap.cc:492] T a8b5ef3087834d89941bf92a2c409563 P e0d74c61684f40338bc384332f90e9c7: Bootstrap starting.
I20260929 22:50:43.497596 22339 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:50:43.497518 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.147.254:45601 (local address 127.14.147.194:53205)
0929 22:50:43.497550 (+ 32us) negotiation.cc:107] Waiting for socket to connect
0929 22:50:43.497572 (+ 22us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.147.254:45601: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":19}
I20260929 22:50:43.497941 22364 tablet_bootstrap.cc:654] T a8b5ef3087834d89941bf92a2c409563 P e0d74c61684f40338bc384332f90e9c7: Neither blocks nor log segments found. Creating new log.
I20260929 22:50:43.498020 22122 heartbeater.cc:503] Master 127.14.147.252:41485 was elected leader, sending a full tablet report...
I20260929 22:50:43.498443 22359 tablet_bootstrap.cc:492] T a8b5ef3087834d89941bf92a2c409563 P f9ed52b3ef574d3d8c17014dbf842a54: No bootstrap required, opened a new log
I20260929 22:50:43.498504 22359 ts_tablet_manager.cc:1397] T a8b5ef3087834d89941bf92a2c409563 P f9ed52b3ef574d3d8c17014dbf842a54: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260929 22:50:43.498590 22364 tablet_bootstrap.cc:492] T a8b5ef3087834d89941bf92a2c409563 P e0d74c61684f40338bc384332f90e9c7: No bootstrap required, opened a new log
I20260929 22:50:43.498638 22359 raft_consensus.cc:348] T a8b5ef3087834d89941bf92a2c409563 P f9ed52b3ef574d3d8c17014dbf842a54 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9ed52b3ef574d3d8c17014dbf842a54" member_type: VOTER last_known_addr { host: "127.14.147.195" port: 38905 } } peers { permanent_uuid: "ab4a15db2c3f422e8646d4b5255cf8df" member_type: VOTER last_known_addr { host: "127.14.147.194" port: 45475 } } peers { permanent_uuid: "e0d74c61684f40338bc384332f90e9c7" member_type: VOTER last_known_addr { host: "127.14.147.193" port: 44967 } }
I20260929 22:50:43.498710 22359 raft_consensus.cc:374] T a8b5ef3087834d89941bf92a2c409563 P f9ed52b3ef574d3d8c17014dbf842a54 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 22:50:43.498739 22359 raft_consensus.cc:729] T a8b5ef3087834d89941bf92a2c409563 P f9ed52b3ef574d3d8c17014dbf842a54 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f9ed52b3ef574d3d8c17014dbf842a54, State: Initialized, Role: FOLLOWER
I20260929 22:50:43.498728 22364 ts_tablet_manager.cc:1397] T a8b5ef3087834d89941bf92a2c409563 P e0d74c61684f40338bc384332f90e9c7: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260929 22:50:43.498785 22359 consensus_queue.cc:260] T a8b5ef3087834d89941bf92a2c409563 P f9ed52b3ef574d3d8c17014dbf842a54 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9ed52b3ef574d3d8c17014dbf842a54" member_type: VOTER last_known_addr { host: "127.14.147.195" port: 38905 } } peers { permanent_uuid: "ab4a15db2c3f422e8646d4b5255cf8df" member_type: VOTER last_known_addr { host: "127.14.147.194" port: 45475 } } peers { permanent_uuid: "e0d74c61684f40338bc384332f90e9c7" member_type: VOTER last_known_addr { host: "127.14.147.193" port: 44967 } }
I20260929 22:50:43.498852 22364 raft_consensus.cc:348] T a8b5ef3087834d89941bf92a2c409563 P e0d74c61684f40338bc384332f90e9c7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9ed52b3ef574d3d8c17014dbf842a54" member_type: VOTER last_known_addr { host: "127.14.147.195" port: 38905 } } peers { permanent_uuid: "ab4a15db2c3f422e8646d4b5255cf8df" member_type: VOTER last_known_addr { host: "127.14.147.194" port: 45475 } } peers { permanent_uuid: "e0d74c61684f40338bc384332f90e9c7" member_type: VOTER last_known_addr { host: "127.14.147.193" port: 44967 } }
I20260929 22:50:43.498908 22364 raft_consensus.cc:374] T a8b5ef3087834d89941bf92a2c409563 P e0d74c61684f40338bc384332f90e9c7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 22:50:43.498924 22364 raft_consensus.cc:729] T a8b5ef3087834d89941bf92a2c409563 P e0d74c61684f40338bc384332f90e9c7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e0d74c61684f40338bc384332f90e9c7, State: Initialized, Role: FOLLOWER
I20260929 22:50:43.498957 22359 ts_tablet_manager.cc:1428] T a8b5ef3087834d89941bf92a2c409563 P f9ed52b3ef574d3d8c17014dbf842a54: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260929 22:50:43.498965 22364 consensus_queue.cc:260] T a8b5ef3087834d89941bf92a2c409563 P e0d74c61684f40338bc384332f90e9c7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9ed52b3ef574d3d8c17014dbf842a54" member_type: VOTER last_known_addr { host: "127.14.147.195" port: 38905 } } peers { permanent_uuid: "ab4a15db2c3f422e8646d4b5255cf8df" member_type: VOTER last_known_addr { host: "127.14.147.194" port: 45475 } } peers { permanent_uuid: "e0d74c61684f40338bc384332f90e9c7" member_type: VOTER last_known_addr { host: "127.14.147.193" port: 44967 } }
I20260929 22:50:43.499125 22364 ts_tablet_manager.cc:1428] T a8b5ef3087834d89941bf92a2c409563 P e0d74c61684f40338bc384332f90e9c7: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260929 22:50:43.499261 22338 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:50:43.499161 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.147.254:45601 (local address 127.14.147.195:49591)
0929 22:50:43.499200 (+ 39us) negotiation.cc:107] Waiting for socket to connect
0929 22:50:43.499234 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.147.254:45601: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":25}
I20260929 22:50:43.499629 22340 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:50:43.499535 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.147.254:45601 (local address 127.14.147.193:37839)
0929 22:50:43.499571 (+ 36us) negotiation.cc:107] Waiting for socket to connect
0929 22:50:43.499603 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.147.254:45601: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":17}
I20260929 22:50:43.499773 22259 heartbeater.cc:503] Master 127.14.147.252:41485 was elected leader, sending a full tablet report...
I20260929 22:50:43.499891 22340 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:50:43.499810 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.147.254:45601 (local address 127.14.147.193:39273)
0929 22:50:43.499843 (+ 33us) negotiation.cc:107] Waiting for socket to connect
0929 22:50:43.499866 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.147.254:45601: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":20}
I20260929 22:50:43.499986 21985 heartbeater.cc:503] Master 127.14.147.252:41485 was elected leader, sending a full tablet report...
I20260929 22:50:43.520895 22363 raft_consensus.cc:482] T a8b5ef3087834d89941bf92a2c409563 P ab4a15db2c3f422e8646d4b5255cf8df [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 22:50:43.520998 22363 raft_consensus.cc:504] T a8b5ef3087834d89941bf92a2c409563 P ab4a15db2c3f422e8646d4b5255cf8df [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9ed52b3ef574d3d8c17014dbf842a54" member_type: VOTER last_known_addr { host: "127.14.147.195" port: 38905 } } peers { permanent_uuid: "ab4a15db2c3f422e8646d4b5255cf8df" member_type: VOTER last_known_addr { host: "127.14.147.194" port: 45475 } } peers { permanent_uuid: "e0d74c61684f40338bc384332f90e9c7" member_type: VOTER last_known_addr { host: "127.14.147.193" port: 44967 } }
I20260929 22:50:43.521344 22363 leader_election.cc:288] T a8b5ef3087834d89941bf92a2c409563 P ab4a15db2c3f422e8646d4b5255cf8df [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f9ed52b3ef574d3d8c17014dbf842a54 (127.14.147.195:38905), e0d74c61684f40338bc384332f90e9c7 (127.14.147.193:44967)
I20260929 22:50:43.524075 22339 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:50:43.521540 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.147.195:38905 (local address 127.14.147.194:47677)
0929 22:50:43.521603 (+ 63us) negotiation.cc:107] Waiting for socket to connect
0929 22:50:43.521616 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 22:50:43.521683 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 22:50:43.521796 (+ 113us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 22:50:43.521797 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 22:50:43.521800 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 22:50:43.521867 (+ 67us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 22:50:43.521869 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 22:50:43.522484 (+ 615us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 22:50:43.522485 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 22:50:43.523261 (+ 776us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 22:50:43.523263 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 22:50:43.523834 (+ 571us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 22:50:43.523836 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 22:50:43.523911 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 22:50:43.523914 (+ 3us) client_negotiation.cc:769] Sending connection context
0929 22:50:43.523957 (+ 43us) client_negotiation.cc:240] Negotiation successful
0929 22:50:43.524000 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":41}
I20260929 22:50:43.524073 22356 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:50:43.521603 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.147.194:47677 (local address 127.14.147.195:38905)
0929 22:50:43.521651 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0929 22:50:43.521653 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 22:50:43.521667 (+ 14us) server_negotiation.cc:407] Connection header received
0929 22:50:43.521739 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 22:50:43.521740 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 22:50:43.521749 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 22:50:43.521794 (+ 45us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 22:50:43.521922 (+ 128us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:50:43.522414 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:50:43.523350 (+ 936us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:50:43.523762 (+ 412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:50:43.523824 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 22:50:43.523834 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0929 22:50:43.523967 (+ 133us) server_negotiation.cc:299] Negotiation successful
0929 22:50:43.524000 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260929 22:50:43.524181 22357 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:50:43.521665 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.147.194:45125 (local address 127.14.147.193:44967)
0929 22:50:43.521721 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0929 22:50:43.521723 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 22:50:43.521927 (+ 204us) server_negotiation.cc:407] Connection header received
0929 22:50:43.521972 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 22:50:43.521973 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 22:50:43.521982 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 22:50:43.522023 (+ 41us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 22:50:43.522149 (+ 126us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:50:43.522606 (+ 457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:50:43.523506 (+ 900us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:50:43.523871 (+ 365us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:50:43.523936 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 22:50:43.523946 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0929 22:50:43.524084 (+ 138us) server_negotiation.cc:299] Negotiation successful
0929 22:50:43.524119 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260929 22:50:43.524197 22368 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:50:43.521540 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.147.193:44967 (local address 127.14.147.194:45125)
0929 22:50:43.521855 (+ 315us) negotiation.cc:107] Waiting for socket to connect
0929 22:50:43.521865 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0929 22:50:43.521923 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 22:50:43.522024 (+ 101us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 22:50:43.522025 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 22:50:43.522026 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 22:50:43.522097 (+ 71us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 22:50:43.522099 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 22:50:43.522675 (+ 576us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 22:50:43.522676 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 22:50:43.523398 (+ 722us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 22:50:43.523400 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 22:50:43.523957 (+ 557us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 22:50:43.523958 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 22:50:43.524026 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 22:50:43.524029 (+ 3us) client_negotiation.cc:769] Sending connection context
0929 22:50:43.524073 (+ 44us) client_negotiation.cc:240] Negotiation successful
0929 22:50:43.524111 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":296,"mutex_wait_us":147,"thread_start_us":113,"threads_started":1}
I20260929 22:50:43.524446 21939 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a8b5ef3087834d89941bf92a2c409563" candidate_uuid: "ab4a15db2c3f422e8646d4b5255cf8df" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e0d74c61684f40338bc384332f90e9c7" is_pre_election: true
I20260929 22:50:43.524480 22213 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a8b5ef3087834d89941bf92a2c409563" candidate_uuid: "ab4a15db2c3f422e8646d4b5255cf8df" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f9ed52b3ef574d3d8c17014dbf842a54" is_pre_election: true
I20260929 22:50:43.524549 21939 raft_consensus.cc:2454] T a8b5ef3087834d89941bf92a2c409563 P e0d74c61684f40338bc384332f90e9c7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab4a15db2c3f422e8646d4b5255cf8df in term 0.
I20260929 22:50:43.524605 22213 raft_consensus.cc:2454] T a8b5ef3087834d89941bf92a2c409563 P f9ed52b3ef574d3d8c17014dbf842a54 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab4a15db2c3f422e8646d4b5255cf8df in term 0.
I20260929 22:50:43.524716 22012 leader_election.cc:302] T a8b5ef3087834d89941bf92a2c409563 P ab4a15db2c3f422e8646d4b5255cf8df [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ab4a15db2c3f422e8646d4b5255cf8df, e0d74c61684f40338bc384332f90e9c7; no voters:
I20260929 22:50:43.524842 22363 raft_consensus.cc:2781] T a8b5ef3087834d89941bf92a2c409563 P ab4a15db2c3f422e8646d4b5255cf8df [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 22:50:43.524892 22363 raft_consensus.cc:482] T a8b5ef3087834d89941bf92a2c409563 P ab4a15db2c3f422e8646d4b5255cf8df [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 22:50:43.524909 22363 raft_consensus.cc:3037] T a8b5ef3087834d89941bf92a2c409563 P ab4a15db2c3f422e8646d4b5255cf8df [term 0 FOLLOWER]: Advancing to term 1
I20260929 22:50:43.525584 22363 raft_consensus.cc:504] T a8b5ef3087834d89941bf92a2c409563 P ab4a15db2c3f422e8646d4b5255cf8df [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9ed52b3ef574d3d8c17014dbf842a54" member_type: VOTER last_known_addr { host: "127.14.147.195" port: 38905 } } peers { permanent_uuid: "ab4a15db2c3f422e8646d4b5255cf8df" member_type: VOTER last_known_addr { host: "127.14.147.194" port: 45475 } } peers { permanent_uuid: "e0d74c61684f40338bc384332f90e9c7" member_type: VOTER last_known_addr { host: "127.14.147.193" port: 44967 } }
I20260929 22:50:43.525704 22363 leader_election.cc:288] T a8b5ef3087834d89941bf92a2c409563 P ab4a15db2c3f422e8646d4b5255cf8df [CANDIDATE]: Term 1 election: Requested vote from peers f9ed52b3ef574d3d8c17014dbf842a54 (127.14.147.195:38905), e0d74c61684f40338bc384332f90e9c7 (127.14.147.193:44967)
I20260929 22:50:43.525837 22213 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a8b5ef3087834d89941bf92a2c409563" candidate_uuid: "ab4a15db2c3f422e8646d4b5255cf8df" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f9ed52b3ef574d3d8c17014dbf842a54"
I20260929 22:50:43.525840 21939 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a8b5ef3087834d89941bf92a2c409563" candidate_uuid: "ab4a15db2c3f422e8646d4b5255cf8df" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e0d74c61684f40338bc384332f90e9c7"
I20260929 22:50:43.525918 22213 raft_consensus.cc:3037] T a8b5ef3087834d89941bf92a2c409563 P f9ed52b3ef574d3d8c17014dbf842a54 [term 0 FOLLOWER]: Advancing to term 1
I20260929 22:50:43.525919 21939 raft_consensus.cc:3037] T a8b5ef3087834d89941bf92a2c409563 P e0d74c61684f40338bc384332f90e9c7 [term 0 FOLLOWER]: Advancing to term 1
I20260929 22:50:43.526679 21939 raft_consensus.cc:2454] T a8b5ef3087834d89941bf92a2c409563 P e0d74c61684f40338bc384332f90e9c7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ab4a15db2c3f422e8646d4b5255cf8df in term 1.
I20260929 22:50:43.526679 22213 raft_consensus.cc:2454] T a8b5ef3087834d89941bf92a2c409563 P f9ed52b3ef574d3d8c17014dbf842a54 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ab4a15db2c3f422e8646d4b5255cf8df in term 1.
I20260929 22:50:43.526844 22010 leader_election.cc:302] T a8b5ef3087834d89941bf92a2c409563 P ab4a15db2c3f422e8646d4b5255cf8df [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ab4a15db2c3f422e8646d4b5255cf8df, f9ed52b3ef574d3d8c17014dbf842a54; no voters:
I20260929 22:50:43.526973 22363 raft_consensus.cc:2781] T a8b5ef3087834d89941bf92a2c409563 P ab4a15db2c3f422e8646d4b5255cf8df [term 1 FOLLOWER]: Leader election won for term 1
I20260929 22:50:43.527128 22363 raft_consensus.cc:686] T a8b5ef3087834d89941bf92a2c409563 P ab4a15db2c3f422e8646d4b5255cf8df [term 1 LEADER]: Becoming Leader. State: Replica: ab4a15db2c3f422e8646d4b5255cf8df, State: Running, Role: LEADER
I20260929 22:50:43.527191 22363 consensus_queue.cc:237] T a8b5ef3087834d89941bf92a2c409563 P ab4a15db2c3f422e8646d4b5255cf8df [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9ed52b3ef574d3d8c17014dbf842a54" member_type: VOTER last_known_addr { host: "127.14.147.195" port: 38905 } } peers { permanent_uuid: "ab4a15db2c3f422e8646d4b5255cf8df" member_type: VOTER last_known_addr { host: "127.14.147.194" port: 45475 } } peers { permanent_uuid: "e0d74c61684f40338bc384332f90e9c7" member_type: VOTER last_known_addr { host: "127.14.147.193" port: 44967 } }
I20260929 22:50:43.527482 22368 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:50:43.527358 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.147.254:45601 (local address 127.14.147.194:48943)
0929 22:50:43.527388 (+ 30us) negotiation.cc:107] Waiting for socket to connect
0929 22:50:43.527449 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.147.254:45601: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":16}
I20260929 22:50:43.528015 21803 catalog_manager.cc:5495] T a8b5ef3087834d89941bf92a2c409563 P ab4a15db2c3f422e8646d4b5255cf8df reported cstate change: term changed from 0 to 1, leader changed from <none> to ab4a15db2c3f422e8646d4b5255cf8df (127.14.147.194). New cstate: current_term: 1 leader_uuid: "ab4a15db2c3f422e8646d4b5255cf8df" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9ed52b3ef574d3d8c17014dbf842a54" member_type: VOTER last_known_addr { host: "127.14.147.195" port: 38905 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ab4a15db2c3f422e8646d4b5255cf8df" member_type: VOTER last_known_addr { host: "127.14.147.194" port: 45475 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e0d74c61684f40338bc384332f90e9c7" member_type: VOTER last_known_addr { host: "127.14.147.193" port: 44967 } health_report { overall_health: UNKNOWN } } }
I20260929 22:50:43.541411 22358 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:50:43.539241 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41236 (local address 127.14.147.194:45475)
0929 22:50:43.539293 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0929 22:50:43.539295 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 22:50:43.539304 (+ 9us) server_negotiation.cc:407] Connection header received
0929 22:50:43.539346 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 22:50:43.539347 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 22:50:43.539366 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 22:50:43.539458 (+ 92us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 22:50:43.539565 (+ 107us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:50:43.540049 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:50:43.540785 (+ 736us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:50:43.540986 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:50:43.541037 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 22:50:43.541135 (+ 98us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 22:50:43.541239 (+ 104us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 22:50:43.541287 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0929 22:50:43.541307 (+ 20us) server_negotiation.cc:299] Negotiation successful
0929 22:50:43.541325 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260929 22:50:43.542541 21939 raft_consensus.cc:1261] T a8b5ef3087834d89941bf92a2c409563 P e0d74c61684f40338bc384332f90e9c7 [term 1 FOLLOWER]: Refusing update from remote peer ab4a15db2c3f422e8646d4b5255cf8df: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 22:50:43.542567 22213 raft_consensus.cc:1261] T a8b5ef3087834d89941bf92a2c409563 P f9ed52b3ef574d3d8c17014dbf842a54 [term 1 FOLLOWER]: Refusing update from remote peer ab4a15db2c3f422e8646d4b5255cf8df: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 22:50:43.542829 22369 consensus_queue.cc:1035] T a8b5ef3087834d89941bf92a2c409563 P ab4a15db2c3f422e8646d4b5255cf8df [LEADER]: Connected to new peer: Peer: permanent_uuid: "e0d74c61684f40338bc384332f90e9c7" member_type: VOTER last_known_addr { host: "127.14.147.193" port: 44967 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 22:50:43.542970 22363 consensus_queue.cc:1035] T a8b5ef3087834d89941bf92a2c409563 P ab4a15db2c3f422e8646d4b5255cf8df [LEADER]: Connected to new peer: Peer: permanent_uuid: "f9ed52b3ef574d3d8c17014dbf842a54" member_type: VOTER last_known_addr { host: "127.14.147.195" port: 38905 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 22:50:43.543797 22340 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:50:43.543685 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.147.254:45601 (local address 127.14.147.193:38669)
0929 22:50:43.543722 (+ 37us) negotiation.cc:107] Waiting for socket to connect
0929 22:50:43.543758 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.147.254:45601: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":19}
I20260929 22:50:43.543888 22338 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:50:43.543785 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.147.254:45601 (local address 127.14.147.195:55869)
0929 22:50:43.543826 (+ 41us) negotiation.cc:107] Waiting for socket to connect
0929 22:50:43.543853 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.147.254:45601: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":26}
I20260929 22:50:43.545126 22368 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:50:43.545018 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.147.254:45601 (local address 127.14.147.194:46965)
0929 22:50:43.545053 (+ 35us) negotiation.cc:107] Waiting for socket to connect
0929 22:50:43.545092 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.147.254:45601: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":20}
I20260929 22:50:43.545250 22338 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:50:43.545135 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.147.254:45601 (local address 127.14.147.195:40963)
0929 22:50:43.545187 (+ 52us) negotiation.cc:107] Waiting for socket to connect
0929 22:50:43.545220 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.147.254:45601: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
I20260929 22:50:43.546690 22368 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:50:43.546575 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.147.254:45601 (local address 127.14.147.194:50125)
0929 22:50:43.546627 (+ 52us) negotiation.cc:107] Waiting for socket to connect
0929 22:50:43.546657 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.147.254:45601: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
I20260929 22:50:43.547204 21804 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60080:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 22:50:43.547253 22340 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:50:43.547160 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.147.254:45601 (local address 127.14.147.193:33741)
0929 22:50:43.547196 (+ 36us) negotiation.cc:107] Waiting for socket to connect
0929 22:50:43.547221 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.147.254:45601: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":24}
I20260929 22:50:43.547291 21804 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60080:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 22:50:43.548697 21804 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 98a8b44ee91645f8a3dc6f02d1c5c698: Sending DeleteTablet for 3 replicas of tablet a8b5ef3087834d89941bf92a2c409563
I20260929 22:50:43.548943 22193 tablet_service.cc:1514] Processing DeleteTablet for tablet a8b5ef3087834d89941bf92a2c409563 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 22:50:43 UTC) from {username='oryx', principal='oryx/127.14.147.252@KRBTEST.COM'} at 127.0.0.1:57304
I20260929 22:50:43.549139 22381 tablet_replica.cc:330] T a8b5ef3087834d89941bf92a2c409563 P f9ed52b3ef574d3d8c17014dbf842a54: stopping tablet replica
I20260929 22:50:43.549188 22381 raft_consensus.cc:2229] T a8b5ef3087834d89941bf92a2c409563 P f9ed52b3ef574d3d8c17014dbf842a54 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 22:50:43.549233 22381 raft_consensus.cc:2258] T a8b5ef3087834d89941bf92a2c409563 P f9ed52b3ef574d3d8c17014dbf842a54 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 22:50:43.549252 22056 tablet_service.cc:1514] Processing DeleteTablet for tablet a8b5ef3087834d89941bf92a2c409563 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 22:50:43 UTC) from {username='oryx', principal='oryx/127.14.147.252@KRBTEST.COM'} at 127.0.0.1:41228
I20260929 22:50:43.549679 14927 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhWOCLk/build/release/bin/kudu with pid 21765
I20260929 22:50:43.550460 22381 ts_tablet_manager.cc:1905] T a8b5ef3087834d89941bf92a2c409563 P f9ed52b3ef574d3d8c17014dbf842a54: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 22:50:43.551994 22381 ts_tablet_manager.cc:1918] T a8b5ef3087834d89941bf92a2c409563 P f9ed52b3ef574d3d8c17014dbf842a54: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 22:50:43.552048 22381 log.cc:1196] T a8b5ef3087834d89941bf92a2c409563 P f9ed52b3ef574d3d8c17014dbf842a54: Deleting WAL directory at /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790722213368738-14927-0/minicluster-data/ts-2/wal/wals/a8b5ef3087834d89941bf92a2c409563
I20260929 22:50:43.552261 22381 ts_tablet_manager.cc:1939] T a8b5ef3087834d89941bf92a2c409563 P f9ed52b3ef574d3d8c17014dbf842a54: Deleting consensus metadata
I20260929 22:50:43.552569 21790 catalog_manager.cc:4841] TS f9ed52b3ef574d3d8c17014dbf842a54 (127.14.147.195:38905): tablet a8b5ef3087834d89941bf92a2c409563 (table test-table [id=5a7b0f1d956a481ab858a187ce88370d]) successfully deleted
I20260929 22:50:43.555569 22382 tablet_replica.cc:330] T a8b5ef3087834d89941bf92a2c409563 P ab4a15db2c3f422e8646d4b5255cf8df: stopping tablet replica
I20260929 22:50:43.555644 22382 raft_consensus.cc:2229] T a8b5ef3087834d89941bf92a2c409563 P ab4a15db2c3f422e8646d4b5255cf8df [term 1 LEADER]: Raft consensus shutting down.
I20260929 22:50:43.555763 22382 raft_consensus.cc:2258] T a8b5ef3087834d89941bf92a2c409563 P ab4a15db2c3f422e8646d4b5255cf8df [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 22:50:43.556047 22382 ts_tablet_manager.cc:1905] T a8b5ef3087834d89941bf92a2c409563 P ab4a15db2c3f422e8646d4b5255cf8df: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260929 22:50:43.557107 21792 connection.cc:535] client connection to 127.14.147.193:44967 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260929 22:50:43.557380 21791 connection.cc:535] server connection from 127.14.147.193:47325 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260929 22:50:43.557526 22382 ts_tablet_manager.cc:1918] T a8b5ef3087834d89941bf92a2c409563 P ab4a15db2c3f422e8646d4b5255cf8df: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 22:50:43.557585 22382 log.cc:1196] T a8b5ef3087834d89941bf92a2c409563 P ab4a15db2c3f422e8646d4b5255cf8df: Deleting WAL directory at /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790722213368738-14927-0/minicluster-data/ts-1/wal/wals/a8b5ef3087834d89941bf92a2c409563
I20260929 22:50:43.557837 22382 ts_tablet_manager.cc:1939] T a8b5ef3087834d89941bf92a2c409563 P ab4a15db2c3f422e8646d4b5255cf8df: Deleting consensus metadata
I20260929 22:50:43.558235 21791 catalog_manager.cc:4841] TS ab4a15db2c3f422e8646d4b5255cf8df (127.14.147.194:45475): tablet a8b5ef3087834d89941bf92a2c409563 (table test-table [id=5a7b0f1d956a481ab858a187ce88370d]) successfully deleted
W20260929 22:50:43.558241 22354 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 22:50:43.557405 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.147.193:44967 (local address 127.0.0.1:53502)
0929 22:50:43.557445 (+ 40us) negotiation.cc:107] Waiting for socket to connect
0929 22:50:43.557457 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0929 22:50:43.557497 (+ 40us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 22:50:43.558201 (+ 704us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.147.193:44967: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":18}
W20260929 22:50:43.558427 21792 catalog_manager.cc:4584] TS e0d74c61684f40338bc384332f90e9c7 (127.14.147.193:44967): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet a8b5ef3087834d89941bf92a2c409563: Network error: Client connection negotiation failed: client connection to 127.14.147.193:44967: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260929 22:50:43.558486 14927 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhWOCLk/build/release/bin/kudu with pid 21993
I20260929 22:50:43.565208 14927 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhWOCLk/build/release/bin/kudu with pid 22130
I20260929 22:50:43.570189 14927 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhWOCLk/build/release/bin/kudu with pid 21481
I20260929 22:50:43.574499 14927 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhWOCLk/build/release/bin/kudu with pid 21671
2026-09-29T22:50:43Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipalMultipleMaster (6209 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790722213368738-14927-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 22:50:43 dist-test-slave-98mk krb5kdc[22385](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 22:50:43 dist-test-slave-98mk krb5kdc[22385](info): set up 2 sockets
Sep 29 22:50:43 dist-test-slave-98mk krb5kdc[22385](info): commencing operation
krb5kdc: starting...
W20260929 22:50:44.520330 14927 mini_kdc.cc:121] Time spent starting KDC: real 0.925s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790722213368738-14927-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790722213368738-14927-0/krb5kdc/test-user.keytab.
Sep 29 22:50:44 dist-test-slave-98mk krb5kdc[22385](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790722244, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T22:50:44Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T22:50:44Z Disabled control of system clock
WARNING: no policy specified for oryx/127.14.147.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.14.147.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.14.147.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790722213368738-14927-0/krb5kdc/oryx_127.14.147.254.keytab.
Entry for principal oryx/127.14.147.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790722213368738-14927-0/krb5kdc/oryx_127.14.147.254.keytab.
I20260929 22:50:44.632064 14927 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhWOCLk/build/release/bin/kudu
/tmp/dist-test-taskhWOCLk/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790722213368738-14927-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790722213368738-14927-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790722213368738-14927-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790722213368738-14927-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.147.254:33005
--webserver_interface=127.14.147.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790722213368738-14927-0/krb5kdc/oryx_127.14.147.254.keytab
--principal=oryx/127.14.147.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.147.212:37887
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.147.254:33005
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790722213368738-14927-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790722213368738-14927-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790722213368738-14927-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 22:50:44.711112 22398 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 22:50:44.711344 22398 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 22:50:44.711366 22398 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 22:50:44.712590 22398 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 22:50:44.712636 22398 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260929 22:50:44.712648 22398 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 22:50:44.712659 22398 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 22:50:44.712677 22398 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 22:50:44.712687 22398 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 22:50:44.714233 22398 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.147.212:37887
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790722213368738-14927-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790722213368738-14927-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.14.147.254:33005
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790722213368738-14927-0/krb5kdc/oryx_127.14.147.254.keytab
--principal=oryx/127.14.147.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.147.254:33005
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790722213368738-14927-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.14.147.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22398
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790722213368738-14927-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 21:58:54 UTC on ccycloud.quasar-xguoti.root.comops.site
build id 4681390
I20260929 22:50:44.714445 22398 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 22:50:44.714622 22398 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 22:50:44.719648 22404 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 22:50:44.719715 22406 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 22:50:44.719659 22403 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 22:50:44.719705 22398 server_base.cc:1034] running on GCE node
I20260929 22:50:44.720137 22398 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 22:50:44.720587 22398 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 22:50:44.721766 22398 hybrid_clock.cc:648] HybridClock initialized: now 1790722244721748 us; error 36 us; skew 500 ppm
Sep 29 22:50:44 dist-test-slave-98mk krb5kdc[22385](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790722244, etypes {rep=17 tkt=17 ses=17}, oryx/127.14.147.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 22:50:44.724289 22398 init.cc:377] Logged in from keytab as oryx/127.14.147.254@KRBTEST.COM (short username oryx)
I20260929 22:50:44.724777 22398 webserver.cc:460] Webserver started at http://127.14.147.254:36103/ using document root <none> and password file <none>
I20260929 22:50:44.724977 22398 fs_manager.cc:359] Metadata directory not provided
I20260929 22:50:44.725023 22398 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 22:50:44.725145 22398 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 22:50:44.726205 22398 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790722213368738-14927-0/minicluster-data/master-0/data/instance:
uuid: "7c99fb093ec947debdad2c01a7fd8027"
format_stamp: "Formatted at 2026-09-29 22:50:44 on dist-test-slave-98mk"
server_key: "e72965a741f1512610f6f008a63c2c4b"
server_key_iv: "1e9516251c5cd87b72337a9aa43bd530"
server_key_version: "encryptionkey@0"
I20260929 22:50:44.726542 22398 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790722213368738-14927-0/minicluster-data/master-0/wal/instance:
uuid: "7c99fb093ec947debdad2c01a7fd8027"
format_stamp: "Formatted at 2026-09-29 22:50:44 on dist-test-slave-98mk"
server_key: "e72965a741f1512610f6f008a63c2c4b"
server_key_iv: "1e9516251c5cd87b72337a9aa43bd530"
server_key_version: "encryptionkey@0"
I20260929 22:50:44.727934 22398 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.002s
I20260929 22:50:44.728978 22413 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 22:50:44.729194 22398 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260929 22:50:44.729300 22398 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790722213368738-14927-0/minicluster-data/master-0/data,/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790722213368738-14927-0/minicluster-data/master-0/wal
uuid: "7c99fb093ec947debdad2c01a7fd8027"
format_stamp: "Formatted at 2026-09-29 22:50:44 on dist-test-slave-98mk"
server_key: "e72965a741f1512610f6f008a63c2c4b"
server_key_iv: "1e9516251c5cd87b72337a9aa43bd530"
server_key_version: "encryptionkey@0"
I20260929 22:50:44.729372 22398 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790722213368738-14927-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790722213368738-14927-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790722213368738-14927-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 22:50:44.745353 22398 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 22:50:44.747515 22398 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 22:50:44.747656 22398 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 22:50:44.752255 22398 rpc_server.cc:307] RPC server started. Bound to: 127.14.147.254:33005
I20260929 22:50:44.752297 22475 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.147.254:33005 every 8 connection(s)
I20260929 22:50:44.752620 22398 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790722213368738-14927-0/minicluster-data/master-0/data/info.pb
I20260929 22:50:44.753300 22476 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 22:50:44.755553 22476 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7c99fb093ec947debdad2c01a7fd8027: Bootstrap starting.
I20260929 22:50:44.756098 22476 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7c99fb093ec947debdad2c01a7fd8027: Neither blocks nor log segments found. Creating new log.
I20260929 22:50:44.756325 22476 log.cc:824] T 00000000000000000000000000000000 P 7c99fb093ec947debdad2c01a7fd8027: Log is configured to *not* fsync() on all Append() calls
I20260929 22:50:44.756707 14927 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhWOCLk/build/release/bin/kudu as pid 22398
I20260929 22:50:44.756794 14927 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790722213368738-14927-0/minicluster-data/master-0/wal/instance
I20260929 22:50:44.756980 14927 external_mini_cluster.cc:1428] Setting key cd034f8d6bdb7b0c3adcda228c160661
I20260929 22:50:44.757026 22476 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7c99fb093ec947debdad2c01a7fd8027: No bootstrap required, opened a new log
I20260929 22:50:44.759724 22476 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7c99fb093ec947debdad2c01a7fd8027 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7c99fb093ec947debdad2c01a7fd8027" member_type: VOTER last_known_addr { host: "127.14.147.254" port: 33005 } }
I20260929 22:50:44.759871 22476 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7c99fb093ec947debdad2c01a7fd8027 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 22:50:44.759902 22476 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7c99fb093ec947debdad2c01a7fd8027 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7c99fb093ec947debdad2c01a7fd8027, State: Initialized, Role: FOLLOWER
I20260929 22:50:44.760020 22476 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7c99fb093ec947debdad2c01a7fd8027 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7c99fb093ec947debdad2c01a7fd8027" member_type: VOTER last_known_addr { host: "127.14.147.254" port: 33005 } }
I20260929 22:50:44.760092 22476 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7c99fb093ec947debdad2c01a7fd8027 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 22:50:44.760130 22476 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7c99fb093ec947debdad2c01a7fd8027 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 22:50:44.760176 22476 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7c99fb093ec947debdad2c01a7fd8027 [term 0 FOLLOWER]: Advancing to term 1
I20260929 22:50:44.760931 22476 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7c99fb093ec947debdad2c01a7fd8027 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7c99fb093ec947debdad2c01a7fd8027" member_type: VOTER last_known_addr { host: "127.14.147.254" port: 33005 } }
I20260929 22:50:44.761045 22476 leader_election.cc:302] T 00000000000000000000000000000000 P 7c99fb093ec947debdad2c01a7fd8027 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7c99fb093ec947debdad2c01a7fd8027; no voters:
I20260929 22:50:44.761233 22476 leader_election.cc:288] T 00000000000000000000000000000000 P 7c99fb093ec947debdad2c01a7fd8027 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 22:50:44.761281 22481 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7c99fb093ec947debdad2c01a7fd8027 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 22:50:44.761445 22481 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7c99fb093ec947debdad2c01a7fd8027 [term 1 LEADER]: Becoming Leader. State: Replica: 7c99fb093ec947debdad2c01a7fd8027, State: Running, Role: LEADER
I20260929 22:50:44.761500 22476 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7c99fb093ec947debdad2c01a7fd8027 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 22:50:44.761576 22481 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7c99fb093ec947debdad2c01a7fd8027 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7c99fb093ec947debdad2c01a7fd8027" member_type: VOTER last_known_addr { host: "127.14.147.254" port: 33005 } }
I20260929 22:50:44.761967 22481 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7c99fb093ec947debdad2c01a7fd8027 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7c99fb093ec947debdad2c01a7fd8027. Latest consensus state: current_term: 1 leader_uuid: "7c99fb093ec947debdad2c01a7fd8027" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7c99fb093ec947debdad2c01a7fd8027" member_type: VOTER last_known_addr { host: "127.14.147.254" port: 33005 } } }
I20260929 22:50:44.762068 22481 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7c99fb093ec947debdad2c01a7fd8027 [sys.catalog]: This master's current role is: LEADER
I20260929 22:50:44.762300 22489 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 22:50:44.762374 22483 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7c99fb093ec947debdad2c01a7fd8027 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7c99fb093ec947debdad2c01a7fd8027" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7c99fb093ec947debdad2c01a7fd8027" member_type: VOTER last_known_addr { host: "127.14.147.254" port: 33005 } } }
I20260929 22:50:44.762435 22483 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7c99fb093ec947debdad2c01a7fd8027 [sys.catalog]: This master's current role is: LEADER
I20260929 22:50:44.762948 22489 catalog_manager.cc:1484] Initializing Kudu cluster ID...
Sep 29 22:50:44 dist-test-slave-98mk krb5kdc[22385](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790722244, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.14.147.254@KRBTEST.COM
I20260929 22:50:44.765141 22489 catalog_manager.cc:1347] Generated new cluster ID: e17dc035e6be4a298c01fed90015f568
I20260929 22:50:44.765193 22489 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 22:50:44.767916 22480 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:50:44.757616 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46490 (local address 127.14.147.254:33005)
0929 22:50:44.757784 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0929 22:50:44.757791 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 22:50:44.757823 (+ 32us) server_negotiation.cc:407] Connection header received
0929 22:50:44.758273 (+ 450us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 22:50:44.758280 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 22:50:44.758337 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 22:50:44.758457 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0929 22:50:44.759233 (+ 776us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:50:44.760222 (+ 989us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:50:44.761228 (+ 1006us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:50:44.761856 (+ 628us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:50:44.761966 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 22:50:44.764787 (+ 2821us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 22:50:44.764792 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 22:50:44.764792 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 22:50:44.764804 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0929 22:50:44.766513 (+ 1709us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 22:50:44.766853 (+ 340us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 22:50:44.766857 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 22:50:44.766857 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 22:50:44.766943 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 22:50:44.767109 (+ 166us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 22:50:44.767110 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 22:50:44.767110 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 22:50:44.767356 (+ 246us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 22:50:44.767541 (+ 185us) server_negotiation.cc:1035] Waiting for connection context
0929 22:50:44.767607 (+ 66us) server_negotiation.cc:299] Negotiation successful
0929 22:50:44.767739 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":118,"thread_start_us":91,"threads_started":1}
I20260929 22:50:44.776391 22489 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 22:50:44.777122 22489 catalog_manager.cc:1504] Loading token signing keys...
I20260929 22:50:44.781396 22489 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7c99fb093ec947debdad2c01a7fd8027: Generated new TSK 0
I20260929 22:50:44.781596 22489 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.14.147.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.14.147.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.14.147.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790722213368738-14927-0/krb5kdc/oryx_127.14.147.193.keytab.
Entry for principal oryx/127.14.147.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790722213368738-14927-0/krb5kdc/oryx_127.14.147.193.keytab.
I20260929 22:50:44.809913 14927 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhWOCLk/build/release/bin/kudu
/tmp/dist-test-taskhWOCLk/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790722213368738-14927-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790722213368738-14927-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790722213368738-14927-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790722213368738-14927-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.147.193:0
--local_ip_for_outbound_sockets=127.14.147.193
--webserver_interface=127.14.147.193
--webserver_port=0
--tserver_master_addrs=127.14.147.254:33005
--keytab_file=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790722213368738-14927-0/krb5kdc/oryx_127.14.147.193.keytab
--principal=oryx/127.14.147.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.147.212:37887
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790722213368738-14927-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790722213368738-14927-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790722213368738-14927-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 22:50:44.899044 22503 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 22:50:44.899267 22503 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 22:50:44.899292 22503 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 22:50:44.900543 22503 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 22:50:44.900590 22503 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 22:50:44.900612 22503 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260929 22:50:44.900629 22503 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.147.193
I20260929 22:50:44.902217 22503 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.147.212:37887
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790722213368738-14927-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790722213368738-14927-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790722213368738-14927-0/krb5kdc/oryx_127.14.147.193.keytab
--principal=oryx/127.14.147.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.147.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790722213368738-14927-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.14.147.193
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.14.147.254:33005
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22503
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.147.193
--log_dir=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790722213368738-14927-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 21:58:54 UTC on ccycloud.quasar-xguoti.root.comops.site
build id 4681390
I20260929 22:50:44.902447 22503 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 22:50:44.902676 22503 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260929 22:50:44.907610 22503 server_base.cc:1034] running on GCE node
W20260929 22:50:44.907581 22509 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 22:50:44.907585 22511 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 22:50:44.907581 22508 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 22:50:44.907990 22503 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 22:50:44.908231 22503 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 22:50:44.909371 22503 hybrid_clock.cc:648] HybridClock initialized: now 1790722244909363 us; error 32 us; skew 500 ppm
Sep 29 22:50:44 dist-test-slave-98mk krb5kdc[22385](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790722244, etypes {rep=17 tkt=17 ses=17}, oryx/127.14.147.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 22:50:44.911895 22503 init.cc:377] Logged in from keytab as oryx/127.14.147.193@KRBTEST.COM (short username oryx)
I20260929 22:50:44.912305 22503 webserver.cc:460] Webserver started at http://127.14.147.193:38579/ using document root <none> and password file <none>
I20260929 22:50:44.912482 22503 fs_manager.cc:359] Metadata directory not provided
I20260929 22:50:44.912520 22503 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 22:50:44.912611 22503 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 22:50:44.913640 22503 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790722213368738-14927-0/minicluster-data/ts-0/data/instance:
uuid: "a46aea30b8a8431189d809e841b82f6e"
format_stamp: "Formatted at 2026-09-29 22:50:44 on dist-test-slave-98mk"
server_key: "34a35f8a62177c8787fa07a4d9f980e7"
server_key_iv: "a536c0651f0f6bd9eceda05d2abfd3b8"
server_key_version: "encryptionkey@0"
I20260929 22:50:44.913947 22503 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790722213368738-14927-0/minicluster-data/ts-0/wal/instance:
uuid: "a46aea30b8a8431189d809e841b82f6e"
format_stamp: "Formatted at 2026-09-29 22:50:44 on dist-test-slave-98mk"
server_key: "34a35f8a62177c8787fa07a4d9f980e7"
server_key_iv: "a536c0651f0f6bd9eceda05d2abfd3b8"
server_key_version: "encryptionkey@0"
I20260929 22:50:44.915161 22503 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260929 22:50:44.915867 22518 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 22:50:44.916043 22503 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260929 22:50:44.916126 22503 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790722213368738-14927-0/minicluster-data/ts-0/data,/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790722213368738-14927-0/minicluster-data/ts-0/wal
uuid: "a46aea30b8a8431189d809e841b82f6e"
format_stamp: "Formatted at 2026-09-29 22:50:44 on dist-test-slave-98mk"
server_key: "34a35f8a62177c8787fa07a4d9f980e7"
server_key_iv: "a536c0651f0f6bd9eceda05d2abfd3b8"
server_key_version: "encryptionkey@0"
I20260929 22:50:44.916208 22503 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790722213368738-14927-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790722213368738-14927-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790722213368738-14927-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 22:50:44.928589 22503 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 22:50:44.930433 22503 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 22:50:44.930573 22503 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 22:50:44.931205 22503 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 22:50:44.931255 22503 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 22:50:44.931284 22503 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 22:50:44.931296 22503 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 29 22:50:44 dist-test-slave-98mk krb5kdc[22385](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790722244, etypes {rep=17 tkt=17 ses=17}, oryx/127.14.147.193@KRBTEST.COM for oryx/127.14.147.254@KRBTEST.COM
I20260929 22:50:44.938089 22503 rpc_server.cc:307] RPC server started. Bound to: 127.14.147.193:33167
I20260929 22:50:44.938150 22633 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.147.193:33167 every 8 connection(s)
I20260929 22:50:44.938457 22503 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790722213368738-14927-0/minicluster-data/ts-0/data/info.pb
I20260929 22:50:44.940781 22480 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:50:44.931618 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.147.193:56337 (local address 127.14.147.254:33005)
0929 22:50:44.931689 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0929 22:50:44.931692 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 22:50:44.931875 (+ 183us) server_negotiation.cc:407] Connection header received
0929 22:50:44.932403 (+ 528us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 22:50:44.932404 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 22:50:44.932417 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 22:50:44.932502 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0929 22:50:44.933203 (+ 701us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:50:44.933802 (+ 599us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:50:44.934820 (+ 1018us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:50:44.935038 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:50:44.935094 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 22:50:44.938253 (+ 3159us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 22:50:44.938254 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 22:50:44.938255 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 22:50:44.938264 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0929 22:50:44.939662 (+ 1398us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 22:50:44.940056 (+ 394us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 22:50:44.940057 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 22:50:44.940057 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 22:50:44.940106 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 22:50:44.940284 (+ 178us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 22:50:44.940285 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 22:50:44.940285 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 22:50:44.940385 (+ 100us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 22:50:44.940433 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0929 22:50:44.940639 (+ 206us) server_negotiation.cc:299] Negotiation successful
0929 22:50:44.940716 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260929 22:50:44.940994 22527 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:50:44.931595 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.147.254:33005 (local address 127.14.147.193:56337)
0929 22:50:44.931748 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0929 22:50:44.931790 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0929 22:50:44.932310 (+ 520us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 22:50:44.932554 (+ 244us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 22:50:44.932558 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 22:50:44.932888 (+ 330us) client_negotiation.cc:189] Negotiated authn=SASL
0929 22:50:44.933107 (+ 219us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 22:50:44.933112 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 22:50:44.933884 (+ 772us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 22:50:44.933885 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 22:50:44.934724 (+ 839us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 22:50:44.934727 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 22:50:44.935110 (+ 383us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 22:50:44.935112 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 22:50:44.935240 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 22:50:44.935812 (+ 572us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 22:50:44.935824 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0929 22:50:44.938149 (+ 2325us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 22:50:44.939768 (+ 1619us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 22:50:44.939774 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 22:50:44.939776 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 22:50:44.939995 (+ 219us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 22:50:44.940148 (+ 153us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 22:50:44.940149 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 22:50:44.940149 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 22:50:44.940242 (+ 93us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 22:50:44.940431 (+ 189us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 22:50:44.940435 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 22:50:44.940552 (+ 117us) client_negotiation.cc:769] Sending connection context
0929 22:50:44.940627 (+ 75us) client_negotiation.cc:240] Negotiation successful
0929 22:50:44.940761 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":82,"threads_started":1}
I20260929 22:50:44.941462 22634 heartbeater.cc:347] Connected to a master server at 127.14.147.254:33005
I20260929 22:50:44.941542 22634 heartbeater.cc:464] Registering TS with master...
I20260929 22:50:44.941716 22634 heartbeater.cc:511] Master 127.14.147.254:33005 requested a full tablet report, sending...
I20260929 22:50:44.942303 22429 ts_manager.cc:194] Registered new tserver with Master: a46aea30b8a8431189d809e841b82f6e (127.14.147.193:33167)
I20260929 22:50:44.943323 22429 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.14.147.193@KRBTEST.COM'} at 127.14.147.193:56337
I20260929 22:50:44.945559 14927 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhWOCLk/build/release/bin/kudu as pid 22503
I20260929 22:50:44.945683 14927 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790722213368738-14927-0/minicluster-data/ts-0/wal/instance
I20260929 22:50:44.945927 14927 external_mini_cluster.cc:1428] Setting key 1e8975a0483d56adadd02d8ef3d3aacd
I20260929 22:50:44.950934 22640 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:50:44.942450 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.147.254:33005 (local address 127.14.147.193:46543)
0929 22:50:44.942589 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0929 22:50:44.942600 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0929 22:50:44.942650 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 22:50:44.942792 (+ 142us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 22:50:44.942793 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 22:50:44.942994 (+ 201us) client_negotiation.cc:189] Negotiated authn=SASL
0929 22:50:44.943064 (+ 70us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 22:50:44.943067 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 22:50:44.943981 (+ 914us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 22:50:44.943981 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 22:50:44.944801 (+ 820us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 22:50:44.944806 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 22:50:44.945321 (+ 515us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 22:50:44.945322 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 22:50:44.945418 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 22:50:44.945990 (+ 572us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 22:50:44.945996 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0929 22:50:44.946412 (+ 416us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 22:50:44.949931 (+ 3519us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 22:50:44.949933 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 22:50:44.949933 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 22:50:44.950136 (+ 203us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 22:50:44.950357 (+ 221us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 22:50:44.950358 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 22:50:44.950358 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 22:50:44.950406 (+ 48us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 22:50:44.950641 (+ 235us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 22:50:44.950641 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 22:50:44.950681 (+ 40us) client_negotiation.cc:769] Sending connection context
0929 22:50:44.950750 (+ 69us) client_negotiation.cc:240] Negotiation successful
0929 22:50:44.950851 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":80,"threads_started":1}
I20260929 22:50:44.950973 22480 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:50:44.942502 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.147.193:46543 (local address 127.14.147.254:33005)
0929 22:50:44.942544 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0929 22:50:44.942546 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 22:50:44.942707 (+ 161us) server_negotiation.cc:407] Connection header received
0929 22:50:44.942728 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 22:50:44.942729 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 22:50:44.942738 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 22:50:44.942783 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0929 22:50:44.943163 (+ 380us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:50:44.943847 (+ 684us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:50:44.944903 (+ 1056us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:50:44.945215 (+ 312us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:50:44.945283 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 22:50:44.946546 (+ 1263us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 22:50:44.946547 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 22:50:44.946548 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 22:50:44.946557 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0929 22:50:44.949851 (+ 3294us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 22:50:44.950231 (+ 380us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 22:50:44.950233 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 22:50:44.950233 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 22:50:44.950298 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 22:50:44.950458 (+ 160us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 22:50:44.950459 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 22:50:44.950459 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 22:50:44.950590 (+ 131us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 22:50:44.950651 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0929 22:50:44.950797 (+ 146us) server_negotiation.cc:299] Negotiation successful
0929 22:50:44.950880 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
WARNING: no policy specified for oryx/127.14.147.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.14.147.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.14.147.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790722213368738-14927-0/krb5kdc/oryx_127.14.147.194.keytab.
Entry for principal oryx/127.14.147.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790722213368738-14927-0/krb5kdc/oryx_127.14.147.194.keytab.
I20260929 22:50:44.974016 14927 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhWOCLk/build/release/bin/kudu
/tmp/dist-test-taskhWOCLk/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790722213368738-14927-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790722213368738-14927-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790722213368738-14927-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790722213368738-14927-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.147.194:0
--local_ip_for_outbound_sockets=127.14.147.194
--webserver_interface=127.14.147.194
--webserver_port=0
--tserver_master_addrs=127.14.147.254:33005
--keytab_file=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790722213368738-14927-0/krb5kdc/oryx_127.14.147.194.keytab
--principal=oryx/127.14.147.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.147.212:37887
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790722213368738-14927-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790722213368738-14927-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790722213368738-14927-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 22:50:45.055208 22643 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 22:50:45.055467 22643 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 22:50:45.055505 22643 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 22:50:45.056732 22643 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 22:50:45.056782 22643 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 22:50:45.056810 22643 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260929 22:50:45.056831 22643 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.147.194
I20260929 22:50:45.058416 22643 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.147.212:37887
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790722213368738-14927-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790722213368738-14927-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790722213368738-14927-0/krb5kdc/oryx_127.14.147.194.keytab
--principal=oryx/127.14.147.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.147.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790722213368738-14927-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.14.147.194
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.14.147.254:33005
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22643
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.147.194
--log_dir=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790722213368738-14927-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 21:58:54 UTC on ccycloud.quasar-xguoti.root.comops.site
build id 4681390
I20260929 22:50:45.058668 22643 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 22:50:45.058897 22643 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260929 22:50:45.063807 22643 server_base.cc:1034] running on GCE node
W20260929 22:50:45.063791 22649 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 22:50:45.063773 22651 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 22:50:45.063786 22648 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 22:50:45.064134 22643 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 22:50:45.064376 22643 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 22:50:45.065528 22643 hybrid_clock.cc:648] HybridClock initialized: now 1790722245065515 us; error 28 us; skew 500 ppm
Sep 29 22:50:45 dist-test-slave-98mk krb5kdc[22385](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790722245, etypes {rep=17 tkt=17 ses=17}, oryx/127.14.147.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 22:50:45.068004 22643 init.cc:377] Logged in from keytab as oryx/127.14.147.194@KRBTEST.COM (short username oryx)
I20260929 22:50:45.068495 22643 webserver.cc:460] Webserver started at http://127.14.147.194:39747/ using document root <none> and password file <none>
I20260929 22:50:45.068728 22643 fs_manager.cc:359] Metadata directory not provided
I20260929 22:50:45.068779 22643 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 22:50:45.068907 22643 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 22:50:45.069963 22643 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790722213368738-14927-0/minicluster-data/ts-1/data/instance:
uuid: "60d7fd130c0f46ea83489a52a291a07a"
format_stamp: "Formatted at 2026-09-29 22:50:45 on dist-test-slave-98mk"
server_key: "e25004a576e9465de281cd9187283e93"
server_key_iv: "2c691dc0043749ea963906785e9e2bd1"
server_key_version: "encryptionkey@0"
I20260929 22:50:45.070320 22643 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790722213368738-14927-0/minicluster-data/ts-1/wal/instance:
uuid: "60d7fd130c0f46ea83489a52a291a07a"
format_stamp: "Formatted at 2026-09-29 22:50:45 on dist-test-slave-98mk"
server_key: "e25004a576e9465de281cd9187283e93"
server_key_iv: "2c691dc0043749ea963906785e9e2bd1"
server_key_version: "encryptionkey@0"
I20260929 22:50:45.071870 22643 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260929 22:50:45.072710 22658 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 22:50:45.072877 22643 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260929 22:50:45.072959 22643 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790722213368738-14927-0/minicluster-data/ts-1/data,/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790722213368738-14927-0/minicluster-data/ts-1/wal
uuid: "60d7fd130c0f46ea83489a52a291a07a"
format_stamp: "Formatted at 2026-09-29 22:50:45 on dist-test-slave-98mk"
server_key: "e25004a576e9465de281cd9187283e93"
server_key_iv: "2c691dc0043749ea963906785e9e2bd1"
server_key_version: "encryptionkey@0"
I20260929 22:50:45.073025 22643 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790722213368738-14927-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790722213368738-14927-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790722213368738-14927-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 22:50:45.086807 22643 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 22:50:45.088776 22643 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 22:50:45.088923 22643 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 22:50:45.089443 22643 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 22:50:45.089483 22643 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 22:50:45.089514 22643 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 22:50:45.089538 22643 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 29 22:50:45 dist-test-slave-98mk krb5kdc[22385](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790722245, etypes {rep=17 tkt=17 ses=17}, oryx/127.14.147.194@KRBTEST.COM for oryx/127.14.147.254@KRBTEST.COM
I20260929 22:50:45.096395 22643 rpc_server.cc:307] RPC server started. Bound to: 127.14.147.194:44459
I20260929 22:50:45.096488 22774 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.147.194:44459 every 8 connection(s)
I20260929 22:50:45.096796 22643 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790722213368738-14927-0/minicluster-data/ts-1/data/info.pb
I20260929 22:50:45.098387 22480 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:50:45.089863 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.147.194:55497 (local address 127.14.147.254:33005)
0929 22:50:45.089924 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0929 22:50:45.089926 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 22:50:45.090220 (+ 294us) server_negotiation.cc:407] Connection header received
0929 22:50:45.090791 (+ 571us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 22:50:45.090793 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 22:50:45.090806 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 22:50:45.090860 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0929 22:50:45.091544 (+ 684us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:50:45.092142 (+ 598us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:50:45.093176 (+ 1034us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:50:45.093362 (+ 186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:50:45.093425 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 22:50:45.095827 (+ 2402us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 22:50:45.095829 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 22:50:45.095829 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 22:50:45.095838 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0929 22:50:45.097351 (+ 1513us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 22:50:45.097688 (+ 337us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 22:50:45.097689 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 22:50:45.097689 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 22:50:45.097736 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 22:50:45.097899 (+ 163us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 22:50:45.097899 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 22:50:45.097900 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 22:50:45.097998 (+ 98us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 22:50:45.098043 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0929 22:50:45.098250 (+ 207us) server_negotiation.cc:299] Negotiation successful
0929 22:50:45.098313 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260929 22:50:45.098592 22667 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:50:45.089887 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.147.254:33005 (local address 127.14.147.194:55497)
0929 22:50:45.090066 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0929 22:50:45.090112 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0929 22:50:45.090678 (+ 566us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 22:50:45.090948 (+ 270us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 22:50:45.090953 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 22:50:45.091216 (+ 263us) client_negotiation.cc:189] Negotiated authn=SASL
0929 22:50:45.091449 (+ 233us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 22:50:45.091456 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 22:50:45.092226 (+ 770us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 22:50:45.092227 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 22:50:45.093087 (+ 860us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 22:50:45.093090 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 22:50:45.093436 (+ 346us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 22:50:45.093437 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 22:50:45.093575 (+ 138us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 22:50:45.094115 (+ 540us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 22:50:45.094126 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0929 22:50:45.095729 (+ 1603us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 22:50:45.097413 (+ 1684us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 22:50:45.097415 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 22:50:45.097417 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 22:50:45.097629 (+ 212us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 22:50:45.097771 (+ 142us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 22:50:45.097772 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 22:50:45.097772 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 22:50:45.097858 (+ 86us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 22:50:45.098034 (+ 176us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 22:50:45.098036 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 22:50:45.098152 (+ 116us) client_negotiation.cc:769] Sending connection context
0929 22:50:45.098234 (+ 82us) client_negotiation.cc:240] Negotiation successful
0929 22:50:45.098366 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":133,"thread_start_us":77,"threads_started":1}
I20260929 22:50:45.099087 22775 heartbeater.cc:347] Connected to a master server at 127.14.147.254:33005
I20260929 22:50:45.099169 22775 heartbeater.cc:464] Registering TS with master...
I20260929 22:50:45.099364 22775 heartbeater.cc:511] Master 127.14.147.254:33005 requested a full tablet report, sending...
I20260929 22:50:45.099794 22430 ts_manager.cc:194] Registered new tserver with Master: 60d7fd130c0f46ea83489a52a291a07a (127.14.147.194:44459)
I20260929 22:50:45.100050 14927 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhWOCLk/build/release/bin/kudu as pid 22643
I20260929 22:50:45.100147 14927 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790722213368738-14927-0/minicluster-data/ts-1/wal/instance
I20260929 22:50:45.100265 22430 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.14.147.194@KRBTEST.COM'} at 127.14.147.194:55497
I20260929 22:50:45.100350 14927 external_mini_cluster.cc:1428] Setting key c87a2e8f5cc36c77c8abe7bbad0214b9
I20260929 22:50:45.106118 22480 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:50:45.100210 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.147.194:52681 (local address 127.14.147.254:33005)
0929 22:50:45.100245 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0929 22:50:45.100247 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 22:50:45.100373 (+ 126us) server_negotiation.cc:407] Connection header received
0929 22:50:45.100474 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 22:50:45.100475 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 22:50:45.100484 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 22:50:45.100510 (+ 26us) server_negotiation.cc:226] Negotiated authn=SASL
0929 22:50:45.100923 (+ 413us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:50:45.101520 (+ 597us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:50:45.102495 (+ 975us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:50:45.102677 (+ 182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:50:45.102733 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 22:50:45.103684 (+ 951us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 22:50:45.103685 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 22:50:45.103686 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 22:50:45.103692 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0929 22:50:45.105019 (+ 1327us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 22:50:45.105389 (+ 370us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 22:50:45.105390 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 22:50:45.105390 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 22:50:45.105447 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 22:50:45.105619 (+ 172us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 22:50:45.105619 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 22:50:45.105619 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 22:50:45.105749 (+ 130us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 22:50:45.105798 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0929 22:50:45.105994 (+ 196us) server_negotiation.cc:299] Negotiation successful
0929 22:50:45.106053 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15}
I20260929 22:50:45.106182 22781 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:50:45.100070 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.147.254:33005 (local address 127.14.147.194:52681)
0929 22:50:45.100315 (+ 245us) negotiation.cc:107] Waiting for socket to connect
0929 22:50:45.100325 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0929 22:50:45.100399 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 22:50:45.100566 (+ 167us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 22:50:45.100567 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 22:50:45.100762 (+ 195us) client_negotiation.cc:189] Negotiated authn=SASL
0929 22:50:45.100827 (+ 65us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 22:50:45.100829 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 22:50:45.101613 (+ 784us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 22:50:45.101613 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 22:50:45.102399 (+ 786us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 22:50:45.102402 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 22:50:45.102746 (+ 344us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 22:50:45.102747 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 22:50:45.102806 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 22:50:45.103224 (+ 418us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 22:50:45.103229 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0929 22:50:45.103614 (+ 385us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 22:50:45.105101 (+ 1487us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 22:50:45.105102 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 22:50:45.105103 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 22:50:45.105322 (+ 219us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 22:50:45.105499 (+ 177us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 22:50:45.105500 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 22:50:45.105501 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 22:50:45.105565 (+ 64us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 22:50:45.105846 (+ 281us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 22:50:45.105847 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 22:50:45.105900 (+ 53us) client_negotiation.cc:769] Sending connection context
0929 22:50:45.105971 (+ 71us) client_negotiation.cc:240] Negotiation successful
0929 22:50:45.106082 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":204,"thread_start_us":69,"threads_started":1}
WARNING: no policy specified for oryx/127.14.147.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.14.147.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.14.147.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790722213368738-14927-0/krb5kdc/oryx_127.14.147.195.keytab.
Entry for principal oryx/127.14.147.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790722213368738-14927-0/krb5kdc/oryx_127.14.147.195.keytab.
I20260929 22:50:45.123361 14927 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhWOCLk/build/release/bin/kudu
/tmp/dist-test-taskhWOCLk/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790722213368738-14927-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790722213368738-14927-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790722213368738-14927-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790722213368738-14927-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.147.195:0
--local_ip_for_outbound_sockets=127.14.147.195
--webserver_interface=127.14.147.195
--webserver_port=0
--tserver_master_addrs=127.14.147.254:33005
--keytab_file=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790722213368738-14927-0/krb5kdc/oryx_127.14.147.195.keytab
--principal=oryx/127.14.147.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.147.212:37887
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790722213368738-14927-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790722213368738-14927-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790722213368738-14927-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 22:50:45.202600 22784 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 22:50:45.202826 22784 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 22:50:45.202862 22784 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 22:50:45.204097 22784 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 22:50:45.204141 22784 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 22:50:45.204169 22784 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260929 22:50:45.204193 22784 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.147.195
I20260929 22:50:45.205827 22784 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.147.212:37887
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790722213368738-14927-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790722213368738-14927-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790722213368738-14927-0/krb5kdc/oryx_127.14.147.195.keytab
--principal=oryx/127.14.147.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.147.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790722213368738-14927-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.14.147.195
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.14.147.254:33005
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22784
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.147.195
--log_dir=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790722213368738-14927-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 21:58:54 UTC on ccycloud.quasar-xguoti.root.comops.site
build id 4681390
I20260929 22:50:45.206082 22784 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 22:50:45.206300 22784 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 22:50:45.210992 22790 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 22:50:45.211117 22784 server_base.cc:1034] running on GCE node
W20260929 22:50:45.210983 22792 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 22:50:45.210994 22789 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 22:50:45.211454 22784 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 22:50:45.211716 22784 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 22:50:45.212862 22784 hybrid_clock.cc:648] HybridClock initialized: now 1790722245212855 us; error 27 us; skew 500 ppm
Sep 29 22:50:45 dist-test-slave-98mk krb5kdc[22385](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790722245, etypes {rep=17 tkt=17 ses=17}, oryx/127.14.147.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 22:50:45.215310 22784 init.cc:377] Logged in from keytab as oryx/127.14.147.195@KRBTEST.COM (short username oryx)
I20260929 22:50:45.215771 22784 webserver.cc:460] Webserver started at http://127.14.147.195:38261/ using document root <none> and password file <none>
I20260929 22:50:45.215982 22784 fs_manager.cc:359] Metadata directory not provided
I20260929 22:50:45.216029 22784 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 22:50:45.216181 22784 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 22:50:45.217509 22784 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790722213368738-14927-0/minicluster-data/ts-2/data/instance:
uuid: "1144095f691a4cac85b7a4890ae1ae78"
format_stamp: "Formatted at 2026-09-29 22:50:45 on dist-test-slave-98mk"
server_key: "295e7c978d15fa83046dc3cbc4158cab"
server_key_iv: "e8bcaf88447578874b8864a9ba1bb210"
server_key_version: "encryptionkey@0"
I20260929 22:50:45.217836 22784 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790722213368738-14927-0/minicluster-data/ts-2/wal/instance:
uuid: "1144095f691a4cac85b7a4890ae1ae78"
format_stamp: "Formatted at 2026-09-29 22:50:45 on dist-test-slave-98mk"
server_key: "295e7c978d15fa83046dc3cbc4158cab"
server_key_iv: "e8bcaf88447578874b8864a9ba1bb210"
server_key_version: "encryptionkey@0"
I20260929 22:50:45.219141 22784 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260929 22:50:45.219887 22799 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 22:50:45.220026 22784 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260929 22:50:45.220109 22784 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790722213368738-14927-0/minicluster-data/ts-2/data,/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790722213368738-14927-0/minicluster-data/ts-2/wal
uuid: "1144095f691a4cac85b7a4890ae1ae78"
format_stamp: "Formatted at 2026-09-29 22:50:45 on dist-test-slave-98mk"
server_key: "295e7c978d15fa83046dc3cbc4158cab"
server_key_iv: "e8bcaf88447578874b8864a9ba1bb210"
server_key_version: "encryptionkey@0"
I20260929 22:50:45.220167 22784 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790722213368738-14927-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790722213368738-14927-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790722213368738-14927-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 22:50:45.235814 22784 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 22:50:45.237919 22784 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 22:50:45.238040 22784 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 22:50:45.238533 22784 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 22:50:45.238571 22784 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 22:50:45.238593 22784 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 22:50:45.238605 22784 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 22:50:45.244853 22784 rpc_server.cc:307] RPC server started. Bound to: 127.14.147.195:35113
I20260929 22:50:45.245020 22915 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.147.195:35113 every 8 connection(s)
I20260929 22:50:45.245237 22784 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790722213368738-14927-0/minicluster-data/ts-2/data/info.pb
Sep 29 22:50:45 dist-test-slave-98mk krb5kdc[22385](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790722245, etypes {rep=17 tkt=17 ses=17}, oryx/127.14.147.195@KRBTEST.COM for oryx/127.14.147.254@KRBTEST.COM
I20260929 22:50:45.248131 14927 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhWOCLk/build/release/bin/kudu as pid 22784
I20260929 22:50:45.248236 14927 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790722213368738-14927-0/minicluster-data/ts-2/wal/instance
I20260929 22:50:45.248281 22480 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:50:45.238961 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.147.195:36083 (local address 127.14.147.254:33005)
0929 22:50:45.239023 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0929 22:50:45.239025 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 22:50:45.239199 (+ 174us) server_negotiation.cc:407] Connection header received
0929 22:50:45.239850 (+ 651us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 22:50:45.239852 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 22:50:45.239865 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 22:50:45.239950 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0929 22:50:45.240787 (+ 837us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:50:45.241577 (+ 790us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:50:45.243172 (+ 1595us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:50:45.243369 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:50:45.243502 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 22:50:45.245970 (+ 2468us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 22:50:45.245972 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 22:50:45.245972 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 22:50:45.245979 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0929 22:50:45.247061 (+ 1082us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 22:50:45.247442 (+ 381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 22:50:45.247443 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 22:50:45.247443 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 22:50:45.247496 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 22:50:45.247692 (+ 196us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 22:50:45.247693 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 22:50:45.247693 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 22:50:45.247792 (+ 99us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 22:50:45.247839 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0929 22:50:45.248104 (+ 265us) server_negotiation.cc:299] Negotiation successful
0929 22:50:45.248189 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260929 22:50:45.248437 22808 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:50:45.238923 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.147.254:33005 (local address 127.14.147.195:36083)
0929 22:50:45.239074 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0929 22:50:45.239115 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0929 22:50:45.239734 (+ 619us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 22:50:45.240031 (+ 297us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 22:50:45.240036 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 22:50:45.240395 (+ 359us) client_negotiation.cc:189] Negotiated authn=SASL
0929 22:50:45.240689 (+ 294us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 22:50:45.240696 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 22:50:45.241691 (+ 995us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 22:50:45.241692 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 22:50:45.243048 (+ 1356us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 22:50:45.243052 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 22:50:45.243523 (+ 471us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 22:50:45.243525 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 22:50:45.243680 (+ 155us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 22:50:45.244290 (+ 610us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 22:50:45.244298 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0929 22:50:45.245864 (+ 1566us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 22:50:45.247132 (+ 1268us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 22:50:45.247138 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 22:50:45.247140 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 22:50:45.247356 (+ 216us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 22:50:45.247547 (+ 191us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 22:50:45.247548 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 22:50:45.247548 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 22:50:45.247647 (+ 99us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 22:50:45.247834 (+ 187us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 22:50:45.247837 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 22:50:45.247958 (+ 121us) client_negotiation.cc:769] Sending connection context
0929 22:50:45.248062 (+ 104us) client_negotiation.cc:240] Negotiation successful
0929 22:50:45.248224 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":81,"threads_started":1}
I20260929 22:50:45.248454 14927 external_mini_cluster.cc:1428] Setting key 037456bda73fd0a92e47e9e1ee3fa681
I20260929 22:50:45.248970 22916 heartbeater.cc:347] Connected to a master server at 127.14.147.254:33005
I20260929 22:50:45.249061 22916 heartbeater.cc:464] Registering TS with master...
I20260929 22:50:45.249406 22916 heartbeater.cc:511] Master 127.14.147.254:33005 requested a full tablet report, sending...
I20260929 22:50:45.249776 22430 ts_manager.cc:194] Registered new tserver with Master: 1144095f691a4cac85b7a4890ae1ae78 (127.14.147.195:35113)
I20260929 22:50:45.250209 22430 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.14.147.195@KRBTEST.COM'} at 127.14.147.195:36083
I20260929 22:50:45.255270 22480 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:50:45.249981 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.147.195:48689 (local address 127.14.147.254:33005)
0929 22:50:45.250016 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0929 22:50:45.250017 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0929 22:50:45.250425 (+ 408us) server_negotiation.cc:407] Connection header received
0929 22:50:45.250463 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 22:50:45.250464 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 22:50:45.250474 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 22:50:45.250527 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0929 22:50:45.250859 (+ 332us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:50:45.251323 (+ 464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:50:45.252020 (+ 697us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:50:45.252214 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:50:45.252266 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 22:50:45.253096 (+ 830us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 22:50:45.253097 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 22:50:45.253098 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 22:50:45.253104 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0929 22:50:45.254298 (+ 1194us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 22:50:45.254592 (+ 294us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 22:50:45.254593 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 22:50:45.254594 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 22:50:45.254647 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 22:50:45.254802 (+ 155us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 22:50:45.254803 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 22:50:45.254803 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 22:50:45.254908 (+ 105us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 22:50:45.254962 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0929 22:50:45.255100 (+ 138us) server_negotiation.cc:299] Negotiation successful
0929 22:50:45.255172 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15}
I20260929 22:50:45.255285 22922 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:50:45.249939 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.147.254:33005 (local address 127.14.147.195:48689)
0929 22:50:45.250354 (+ 415us) negotiation.cc:107] Waiting for socket to connect
0929 22:50:45.250366 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0929 22:50:45.250423 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 22:50:45.250571 (+ 148us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 22:50:45.250572 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 22:50:45.250730 (+ 158us) client_negotiation.cc:189] Negotiated authn=SASL
0929 22:50:45.250790 (+ 60us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 22:50:45.250791 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 22:50:45.251434 (+ 643us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 22:50:45.251435 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 22:50:45.251952 (+ 517us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 22:50:45.251953 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 22:50:45.252272 (+ 319us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 22:50:45.252273 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 22:50:45.252340 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 22:50:45.252734 (+ 394us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 22:50:45.252739 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0929 22:50:45.253017 (+ 278us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 22:50:45.254374 (+ 1357us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 22:50:45.254375 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 22:50:45.254376 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 22:50:45.254532 (+ 156us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 22:50:45.254699 (+ 167us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 22:50:45.254699 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 22:50:45.254700 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 22:50:45.254755 (+ 55us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 22:50:45.254962 (+ 207us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 22:50:45.254963 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 22:50:45.255009 (+ 46us) client_negotiation.cc:769] Sending connection context
0929 22:50:45.255070 (+ 61us) client_negotiation.cc:240] Negotiation successful
0929 22:50:45.255175 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":387,"thread_start_us":61,"threads_started":1}
I20260929 22:50:45.260106 14927 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 29 22:50:45 dist-test-slave-98mk krb5kdc[22385](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790722245, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 29 22:50:45 dist-test-slave-98mk krb5kdc[22385](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.14.147.254@KRBTEST.COM, Server not found in Kerberos database
Sep 29 22:50:45 dist-test-slave-98mk krb5kdc[22385](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.14.147.254@KRBTEST.COM, Server not found in Kerberos database
W20260929 22:50:45.280573 22480 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 22:50:45.275059 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46502 (local address 127.14.147.254:33005)
0929 22:50:45.275109 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0929 22:50:45.275110 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0929 22:50:45.275118 (+ 8us) server_negotiation.cc:407] Connection header received
0929 22:50:45.275143 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 22:50:45.275143 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 22:50:45.275152 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 22:50:45.275208 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0929 22:50:45.275772 (+ 564us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:50:45.276324 (+ 552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:50:45.277141 (+ 817us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:50:45.277324 (+ 183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:50:45.277388 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 22:50:45.280507 (+ 3119us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:46502: BlockingRecv error: recv got EOF from 127.0.0.1:46502 (error 108)
Metrics: {"server-negotiator.queue_time_us":28}
Sep 29 22:50:45 dist-test-slave-98mk krb5kdc[22385](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790722245, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for oryx/127.14.147.254@KRBTEST.COM
I20260929 22:50:45.287751 22480 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:50:45.281341 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46512 (local address 127.14.147.254:33005)
0929 22:50:45.281407 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0929 22:50:45.281408 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0929 22:50:45.281510 (+ 102us) server_negotiation.cc:407] Connection header received
0929 22:50:45.281558 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 22:50:45.281559 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 22:50:45.281570 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 22:50:45.281621 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0929 22:50:45.282066 (+ 445us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:50:45.282544 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:50:45.283198 (+ 654us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:50:45.283388 (+ 190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:50:45.283495 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 22:50:45.285525 (+ 2030us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 22:50:45.285526 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 22:50:45.285527 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 22:50:45.285534 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0929 22:50:45.286745 (+ 1211us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 22:50:45.287050 (+ 305us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 22:50:45.287051 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 22:50:45.287051 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 22:50:45.287101 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 22:50:45.287257 (+ 156us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 22:50:45.287258 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 22:50:45.287258 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 22:50:45.287368 (+ 110us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 22:50:45.287484 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0929 22:50:45.287579 (+ 95us) server_negotiation.cc:299] Negotiation successful
0929 22:50:45.287654 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260929 22:50:45.289274 22430 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:46512:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 22:50:45.290979 22430 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 22:50:45.297770 22939 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:50:45.293707 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.147.193:33167 (local address 127.0.0.1:41272)
0929 22:50:45.293984 (+ 277us) negotiation.cc:107] Waiting for socket to connect
0929 22:50:45.294026 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0929 22:50:45.294088 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 22:50:45.294387 (+ 299us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 22:50:45.294390 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 22:50:45.294393 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 22:50:45.294481 (+ 88us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 22:50:45.294484 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 22:50:45.295818 (+ 1334us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 22:50:45.295819 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 22:50:45.296747 (+ 928us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 22:50:45.296750 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 22:50:45.297455 (+ 705us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 22:50:45.297457 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 22:50:45.297535 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 22:50:45.297548 (+ 13us) client_negotiation.cc:769] Sending connection context
0929 22:50:45.297611 (+ 63us) client_negotiation.cc:240] Negotiation successful
0929 22:50:45.297664 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":242,"mutex_wait_us":39,"thread_start_us":46,"threads_started":1}
I20260929 22:50:45.297860 22941 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:50:45.293807 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49068 (local address 127.14.147.195:35113)
0929 22:50:45.294382 (+ 575us) server_negotiation.cc:206] Beginning negotiation
0929 22:50:45.294386 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 22:50:45.294395 (+ 9us) server_negotiation.cc:407] Connection header received
0929 22:50:45.294428 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 22:50:45.294431 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 22:50:45.294443 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 22:50:45.294510 (+ 67us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 22:50:45.294642 (+ 132us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:50:45.295368 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:50:45.296723 (+ 1355us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:50:45.297445 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:50:45.297532 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 22:50:45.297583 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0929 22:50:45.297716 (+ 133us) server_negotiation.cc:299] Negotiation successful
0929 22:50:45.297759 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":537,"thread_start_us":91,"threads_started":1}
I20260929 22:50:45.298074 22942 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:50:45.293892 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.147.194:44459 (local address 127.0.0.1:46792)
0929 22:50:45.294212 (+ 320us) negotiation.cc:107] Waiting for socket to connect
0929 22:50:45.294220 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0929 22:50:45.294258 (+ 38us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 22:50:45.294593 (+ 335us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 22:50:45.294593 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 22:50:45.294596 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 22:50:45.294627 (+ 31us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 22:50:45.294628 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 22:50:45.295846 (+ 1218us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 22:50:45.295847 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 22:50:45.296904 (+ 1057us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 22:50:45.296906 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 22:50:45.297857 (+ 951us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 22:50:45.297859 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 22:50:45.297917 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 22:50:45.297922 (+ 5us) client_negotiation.cc:769] Sending connection context
0929 22:50:45.297970 (+ 48us) client_negotiation.cc:240] Negotiation successful
0929 22:50:45.298008 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":289,"thread_start_us":75,"threads_started":1}
I20260929 22:50:45.297778 22938 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:50:45.293716 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.147.195:35113 (local address 127.0.0.1:49068)
0929 22:50:45.294013 (+ 297us) negotiation.cc:107] Waiting for socket to connect
0929 22:50:45.294029 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 22:50:45.294081 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 22:50:45.294510 (+ 429us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 22:50:45.294512 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 22:50:45.294515 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 22:50:45.294567 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 22:50:45.294568 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 22:50:45.295464 (+ 896us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 22:50:45.295466 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 22:50:45.296634 (+ 1168us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 22:50:45.296637 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 22:50:45.297569 (+ 932us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 22:50:45.297569 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 22:50:45.297639 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 22:50:45.297642 (+ 3us) client_negotiation.cc:769] Sending connection context
0929 22:50:45.297690 (+ 48us) client_negotiation.cc:240] Negotiation successful
0929 22:50:45.297723 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":284,"thread_start_us":86,"threads_started":1}
I20260929 22:50:45.298326 22943 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:50:45.294033 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46792 (local address 127.14.147.194:44459)
0929 22:50:45.294247 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0929 22:50:45.294251 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 22:50:45.294262 (+ 11us) server_negotiation.cc:407] Connection header received
0929 22:50:45.294500 (+ 238us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 22:50:45.294503 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 22:50:45.294520 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 22:50:45.294596 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 22:50:45.294678 (+ 82us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:50:45.295770 (+ 1092us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:50:45.296982 (+ 1212us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:50:45.297692 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:50:45.297772 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 22:50:45.297822 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0929 22:50:45.298170 (+ 348us) server_negotiation.cc:299] Negotiation successful
0929 22:50:45.298226 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":177,"thread_start_us":80,"threads_started":1}
I20260929 22:50:45.298372 22940 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:50:45.293823 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41272 (local address 127.14.147.193:33167)
0929 22:50:45.294221 (+ 398us) server_negotiation.cc:206] Beginning negotiation
0929 22:50:45.294226 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 22:50:45.294238 (+ 12us) server_negotiation.cc:407] Connection header received
0929 22:50:45.294280 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 22:50:45.294282 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 22:50:45.294297 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 22:50:45.294383 (+ 86us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 22:50:45.294568 (+ 185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:50:45.295719 (+ 1151us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:50:45.296826 (+ 1107us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:50:45.297379 (+ 553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:50:45.297459 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 22:50:45.297521 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0929 22:50:45.297827 (+ 306us) server_negotiation.cc:299] Negotiation successful
0929 22:50:45.298273 (+ 446us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":364,"thread_start_us":62,"threads_started":1}
I20260929 22:50:45.298996 22568 tablet_service.cc:1467] Processing CreateTablet for tablet f4e8fbb291994aa88233224f12f35a57 (DEFAULT_TABLE table=test-table [id=2dfa0c77190741728651d32c95aef52f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 22:50:45.298996 22835 tablet_service.cc:1467] Processing CreateTablet for tablet f4e8fbb291994aa88233224f12f35a57 (DEFAULT_TABLE table=test-table [id=2dfa0c77190741728651d32c95aef52f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 22:50:45.299394 22568 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f4e8fbb291994aa88233224f12f35a57. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 22:50:45.299394 22835 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f4e8fbb291994aa88233224f12f35a57. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 22:50:45.299778 22709 tablet_service.cc:1467] Processing CreateTablet for tablet f4e8fbb291994aa88233224f12f35a57 (DEFAULT_TABLE table=test-table [id=2dfa0c77190741728651d32c95aef52f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 22:50:45.300079 22709 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f4e8fbb291994aa88233224f12f35a57. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 22:50:45.303114 22944 tablet_bootstrap.cc:492] T f4e8fbb291994aa88233224f12f35a57 P 1144095f691a4cac85b7a4890ae1ae78: Bootstrap starting.
I20260929 22:50:45.303462 22945 tablet_bootstrap.cc:492] T f4e8fbb291994aa88233224f12f35a57 P a46aea30b8a8431189d809e841b82f6e: Bootstrap starting.
I20260929 22:50:45.303586 22946 tablet_bootstrap.cc:492] T f4e8fbb291994aa88233224f12f35a57 P 60d7fd130c0f46ea83489a52a291a07a: Bootstrap starting.
I20260929 22:50:45.304131 22944 tablet_bootstrap.cc:654] T f4e8fbb291994aa88233224f12f35a57 P 1144095f691a4cac85b7a4890ae1ae78: Neither blocks nor log segments found. Creating new log.
I20260929 22:50:45.304208 22945 tablet_bootstrap.cc:654] T f4e8fbb291994aa88233224f12f35a57 P a46aea30b8a8431189d809e841b82f6e: Neither blocks nor log segments found. Creating new log.
I20260929 22:50:45.304208 22946 tablet_bootstrap.cc:654] T f4e8fbb291994aa88233224f12f35a57 P 60d7fd130c0f46ea83489a52a291a07a: Neither blocks nor log segments found. Creating new log.
I20260929 22:50:45.304430 22944 log.cc:824] T f4e8fbb291994aa88233224f12f35a57 P 1144095f691a4cac85b7a4890ae1ae78: Log is configured to *not* fsync() on all Append() calls
I20260929 22:50:45.304556 22946 log.cc:824] T f4e8fbb291994aa88233224f12f35a57 P 60d7fd130c0f46ea83489a52a291a07a: Log is configured to *not* fsync() on all Append() calls
I20260929 22:50:45.304654 22945 log.cc:824] T f4e8fbb291994aa88233224f12f35a57 P a46aea30b8a8431189d809e841b82f6e: Log is configured to *not* fsync() on all Append() calls
I20260929 22:50:45.305264 22946 tablet_bootstrap.cc:492] T f4e8fbb291994aa88233224f12f35a57 P 60d7fd130c0f46ea83489a52a291a07a: No bootstrap required, opened a new log
I20260929 22:50:45.305272 22944 tablet_bootstrap.cc:492] T f4e8fbb291994aa88233224f12f35a57 P 1144095f691a4cac85b7a4890ae1ae78: No bootstrap required, opened a new log
I20260929 22:50:45.305339 22946 ts_tablet_manager.cc:1397] T f4e8fbb291994aa88233224f12f35a57 P 60d7fd130c0f46ea83489a52a291a07a: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260929 22:50:45.305339 22944 ts_tablet_manager.cc:1397] T f4e8fbb291994aa88233224f12f35a57 P 1144095f691a4cac85b7a4890ae1ae78: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260929 22:50:45.305359 22945 tablet_bootstrap.cc:492] T f4e8fbb291994aa88233224f12f35a57 P a46aea30b8a8431189d809e841b82f6e: No bootstrap required, opened a new log
I20260929 22:50:45.305415 22945 ts_tablet_manager.cc:1397] T f4e8fbb291994aa88233224f12f35a57 P a46aea30b8a8431189d809e841b82f6e: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260929 22:50:45.306802 22945 raft_consensus.cc:348] T f4e8fbb291994aa88233224f12f35a57 P a46aea30b8a8431189d809e841b82f6e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a46aea30b8a8431189d809e841b82f6e" member_type: VOTER last_known_addr { host: "127.14.147.193" port: 33167 } } peers { permanent_uuid: "1144095f691a4cac85b7a4890ae1ae78" member_type: VOTER last_known_addr { host: "127.14.147.195" port: 35113 } } peers { permanent_uuid: "60d7fd130c0f46ea83489a52a291a07a" member_type: VOTER last_known_addr { host: "127.14.147.194" port: 44459 } }
I20260929 22:50:45.306955 22945 raft_consensus.cc:374] T f4e8fbb291994aa88233224f12f35a57 P a46aea30b8a8431189d809e841b82f6e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 22:50:45.306988 22945 raft_consensus.cc:729] T f4e8fbb291994aa88233224f12f35a57 P a46aea30b8a8431189d809e841b82f6e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a46aea30b8a8431189d809e841b82f6e, State: Initialized, Role: FOLLOWER
I20260929 22:50:45.307097 22945 consensus_queue.cc:260] T f4e8fbb291994aa88233224f12f35a57 P a46aea30b8a8431189d809e841b82f6e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a46aea30b8a8431189d809e841b82f6e" member_type: VOTER last_known_addr { host: "127.14.147.193" port: 33167 } } peers { permanent_uuid: "1144095f691a4cac85b7a4890ae1ae78" member_type: VOTER last_known_addr { host: "127.14.147.195" port: 35113 } } peers { permanent_uuid: "60d7fd130c0f46ea83489a52a291a07a" member_type: VOTER last_known_addr { host: "127.14.147.194" port: 44459 } }
I20260929 22:50:45.307384 22945 ts_tablet_manager.cc:1428] T f4e8fbb291994aa88233224f12f35a57 P a46aea30b8a8431189d809e841b82f6e: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260929 22:50:45.307314 22944 raft_consensus.cc:348] T f4e8fbb291994aa88233224f12f35a57 P 1144095f691a4cac85b7a4890ae1ae78 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a46aea30b8a8431189d809e841b82f6e" member_type: VOTER last_known_addr { host: "127.14.147.193" port: 33167 } } peers { permanent_uuid: "1144095f691a4cac85b7a4890ae1ae78" member_type: VOTER last_known_addr { host: "127.14.147.195" port: 35113 } } peers { permanent_uuid: "60d7fd130c0f46ea83489a52a291a07a" member_type: VOTER last_known_addr { host: "127.14.147.194" port: 44459 } }
I20260929 22:50:45.307324 22946 raft_consensus.cc:348] T f4e8fbb291994aa88233224f12f35a57 P 60d7fd130c0f46ea83489a52a291a07a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a46aea30b8a8431189d809e841b82f6e" member_type: VOTER last_known_addr { host: "127.14.147.193" port: 33167 } } peers { permanent_uuid: "1144095f691a4cac85b7a4890ae1ae78" member_type: VOTER last_known_addr { host: "127.14.147.195" port: 35113 } } peers { permanent_uuid: "60d7fd130c0f46ea83489a52a291a07a" member_type: VOTER last_known_addr { host: "127.14.147.194" port: 44459 } }
I20260929 22:50:45.307502 22946 raft_consensus.cc:374] T f4e8fbb291994aa88233224f12f35a57 P 60d7fd130c0f46ea83489a52a291a07a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 22:50:45.307502 22944 raft_consensus.cc:374] T f4e8fbb291994aa88233224f12f35a57 P 1144095f691a4cac85b7a4890ae1ae78 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 22:50:45.307538 22634 heartbeater.cc:503] Master 127.14.147.254:33005 was elected leader, sending a full tablet report...
I20260929 22:50:45.307538 22946 raft_consensus.cc:729] T f4e8fbb291994aa88233224f12f35a57 P 60d7fd130c0f46ea83489a52a291a07a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 60d7fd130c0f46ea83489a52a291a07a, State: Initialized, Role: FOLLOWER
I20260929 22:50:45.307538 22944 raft_consensus.cc:729] T f4e8fbb291994aa88233224f12f35a57 P 1144095f691a4cac85b7a4890ae1ae78 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1144095f691a4cac85b7a4890ae1ae78, State: Initialized, Role: FOLLOWER
I20260929 22:50:45.307644 22946 consensus_queue.cc:260] T f4e8fbb291994aa88233224f12f35a57 P 60d7fd130c0f46ea83489a52a291a07a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a46aea30b8a8431189d809e841b82f6e" member_type: VOTER last_known_addr { host: "127.14.147.193" port: 33167 } } peers { permanent_uuid: "1144095f691a4cac85b7a4890ae1ae78" member_type: VOTER last_known_addr { host: "127.14.147.195" port: 35113 } } peers { permanent_uuid: "60d7fd130c0f46ea83489a52a291a07a" member_type: VOTER last_known_addr { host: "127.14.147.194" port: 44459 } }
I20260929 22:50:45.307693 22944 consensus_queue.cc:260] T f4e8fbb291994aa88233224f12f35a57 P 1144095f691a4cac85b7a4890ae1ae78 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a46aea30b8a8431189d809e841b82f6e" member_type: VOTER last_known_addr { host: "127.14.147.193" port: 33167 } } peers { permanent_uuid: "1144095f691a4cac85b7a4890ae1ae78" member_type: VOTER last_known_addr { host: "127.14.147.195" port: 35113 } } peers { permanent_uuid: "60d7fd130c0f46ea83489a52a291a07a" member_type: VOTER last_known_addr { host: "127.14.147.194" port: 44459 } }
I20260929 22:50:45.307901 22946 ts_tablet_manager.cc:1428] T f4e8fbb291994aa88233224f12f35a57 P 60d7fd130c0f46ea83489a52a291a07a: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260929 22:50:45.307909 22944 ts_tablet_manager.cc:1428] T f4e8fbb291994aa88233224f12f35a57 P 1144095f691a4cac85b7a4890ae1ae78: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260929 22:50:45.307981 22775 heartbeater.cc:503] Master 127.14.147.254:33005 was elected leader, sending a full tablet report...
I20260929 22:50:45.308027 22916 heartbeater.cc:503] Master 127.14.147.254:33005 was elected leader, sending a full tablet report...
I20260929 22:50:45.335232 22952 raft_consensus.cc:482] T f4e8fbb291994aa88233224f12f35a57 P 1144095f691a4cac85b7a4890ae1ae78 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 22:50:45.335362 22952 raft_consensus.cc:504] T f4e8fbb291994aa88233224f12f35a57 P 1144095f691a4cac85b7a4890ae1ae78 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a46aea30b8a8431189d809e841b82f6e" member_type: VOTER last_known_addr { host: "127.14.147.193" port: 33167 } } peers { permanent_uuid: "1144095f691a4cac85b7a4890ae1ae78" member_type: VOTER last_known_addr { host: "127.14.147.195" port: 35113 } } peers { permanent_uuid: "60d7fd130c0f46ea83489a52a291a07a" member_type: VOTER last_known_addr { host: "127.14.147.194" port: 44459 } }
I20260929 22:50:45.335749 22952 leader_election.cc:288] T f4e8fbb291994aa88233224f12f35a57 P 1144095f691a4cac85b7a4890ae1ae78 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a46aea30b8a8431189d809e841b82f6e (127.14.147.193:33167), 60d7fd130c0f46ea83489a52a291a07a (127.14.147.194:44459)
I20260929 22:50:45.338513 22943 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:50:45.336102 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.147.195:34681 (local address 127.14.147.194:44459)
0929 22:50:45.336162 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0929 22:50:45.336164 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 22:50:45.336231 (+ 67us) server_negotiation.cc:407] Connection header received
0929 22:50:45.336262 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 22:50:45.336263 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 22:50:45.336271 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 22:50:45.336322 (+ 51us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 22:50:45.336477 (+ 155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:50:45.336937 (+ 460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:50:45.337830 (+ 893us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:50:45.338200 (+ 370us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:50:45.338262 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 22:50:45.338270 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0929 22:50:45.338412 (+ 142us) server_negotiation.cc:299] Negotiation successful
0929 22:50:45.338449 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260929 22:50:45.338519 22940 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:50:45.336001 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.147.195:46383 (local address 127.14.147.193:33167)
0929 22:50:45.336051 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0929 22:50:45.336052 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0929 22:50:45.336065 (+ 13us) server_negotiation.cc:407] Connection header received
0929 22:50:45.336096 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 22:50:45.336097 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 22:50:45.336106 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 22:50:45.336172 (+ 66us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 22:50:45.336360 (+ 188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:50:45.336869 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:50:45.337826 (+ 957us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:50:45.338183 (+ 357us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:50:45.338239 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 22:50:45.338248 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0929 22:50:45.338421 (+ 173us) server_negotiation.cc:299] Negotiation successful
0929 22:50:45.338454 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260929 22:50:45.338532 22953 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:50:45.336019 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.147.194:44459 (local address 127.14.147.195:34681)
0929 22:50:45.336182 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0929 22:50:45.336190 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0929 22:50:45.336228 (+ 38us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 22:50:45.336330 (+ 102us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 22:50:45.336330 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 22:50:45.336332 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 22:50:45.336412 (+ 80us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 22:50:45.336413 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 22:50:45.337011 (+ 598us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 22:50:45.337011 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 22:50:45.337741 (+ 730us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 22:50:45.337743 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 22:50:45.338277 (+ 534us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 22:50:45.338278 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 22:50:45.338343 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 22:50:45.338347 (+ 4us) client_negotiation.cc:769] Sending connection context
0929 22:50:45.338392 (+ 45us) client_negotiation.cc:240] Negotiation successful
0929 22:50:45.338435 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":146,"mutex_wait_us":11,"thread_start_us":108,"threads_started":1}
I20260929 22:50:45.338536 22808 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:50:45.335911 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.147.193:33167 (local address 127.14.147.195:46383)
0929 22:50:45.335982 (+ 71us) negotiation.cc:107] Waiting for socket to connect
0929 22:50:45.335994 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0929 22:50:45.336041 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 22:50:45.336204 (+ 163us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 22:50:45.336206 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 22:50:45.336207 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 22:50:45.336282 (+ 75us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 22:50:45.336284 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 22:50:45.336941 (+ 657us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 22:50:45.336941 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 22:50:45.337740 (+ 799us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 22:50:45.337742 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 22:50:45.338254 (+ 512us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 22:50:45.338255 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 22:50:45.338348 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 22:50:45.338352 (+ 4us) client_negotiation.cc:769] Sending connection context
0929 22:50:45.338402 (+ 50us) client_negotiation.cc:240] Negotiation successful
0929 22:50:45.338444 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":40,"mutex_wait_us":3}
I20260929 22:50:45.338894 22729 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f4e8fbb291994aa88233224f12f35a57" candidate_uuid: "1144095f691a4cac85b7a4890ae1ae78" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "60d7fd130c0f46ea83489a52a291a07a" is_pre_election: true
I20260929 22:50:45.339073 22729 raft_consensus.cc:2454] T f4e8fbb291994aa88233224f12f35a57 P 60d7fd130c0f46ea83489a52a291a07a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1144095f691a4cac85b7a4890ae1ae78 in term 0.
I20260929 22:50:45.339033 22585 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f4e8fbb291994aa88233224f12f35a57" candidate_uuid: "1144095f691a4cac85b7a4890ae1ae78" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a46aea30b8a8431189d809e841b82f6e" is_pre_election: true
I20260929 22:50:45.339246 22585 raft_consensus.cc:2454] T f4e8fbb291994aa88233224f12f35a57 P a46aea30b8a8431189d809e841b82f6e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1144095f691a4cac85b7a4890ae1ae78 in term 0.
I20260929 22:50:45.339601 22803 leader_election.cc:302] T f4e8fbb291994aa88233224f12f35a57 P 1144095f691a4cac85b7a4890ae1ae78 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1144095f691a4cac85b7a4890ae1ae78, 60d7fd130c0f46ea83489a52a291a07a; no voters:
I20260929 22:50:45.339735 22952 raft_consensus.cc:2781] T f4e8fbb291994aa88233224f12f35a57 P 1144095f691a4cac85b7a4890ae1ae78 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 22:50:45.339803 22952 raft_consensus.cc:482] T f4e8fbb291994aa88233224f12f35a57 P 1144095f691a4cac85b7a4890ae1ae78 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 22:50:45.339826 22952 raft_consensus.cc:3037] T f4e8fbb291994aa88233224f12f35a57 P 1144095f691a4cac85b7a4890ae1ae78 [term 0 FOLLOWER]: Advancing to term 1
I20260929 22:50:45.340791 22952 raft_consensus.cc:504] T f4e8fbb291994aa88233224f12f35a57 P 1144095f691a4cac85b7a4890ae1ae78 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a46aea30b8a8431189d809e841b82f6e" member_type: VOTER last_known_addr { host: "127.14.147.193" port: 33167 } } peers { permanent_uuid: "1144095f691a4cac85b7a4890ae1ae78" member_type: VOTER last_known_addr { host: "127.14.147.195" port: 35113 } } peers { permanent_uuid: "60d7fd130c0f46ea83489a52a291a07a" member_type: VOTER last_known_addr { host: "127.14.147.194" port: 44459 } }
I20260929 22:50:45.340905 22952 leader_election.cc:288] T f4e8fbb291994aa88233224f12f35a57 P 1144095f691a4cac85b7a4890ae1ae78 [CANDIDATE]: Term 1 election: Requested vote from peers a46aea30b8a8431189d809e841b82f6e (127.14.147.193:33167), 60d7fd130c0f46ea83489a52a291a07a (127.14.147.194:44459)
I20260929 22:50:45.341060 22585 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f4e8fbb291994aa88233224f12f35a57" candidate_uuid: "1144095f691a4cac85b7a4890ae1ae78" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a46aea30b8a8431189d809e841b82f6e"
I20260929 22:50:45.341087 22729 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f4e8fbb291994aa88233224f12f35a57" candidate_uuid: "1144095f691a4cac85b7a4890ae1ae78" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "60d7fd130c0f46ea83489a52a291a07a"
I20260929 22:50:45.341153 22729 raft_consensus.cc:3037] T f4e8fbb291994aa88233224f12f35a57 P 60d7fd130c0f46ea83489a52a291a07a [term 0 FOLLOWER]: Advancing to term 1
I20260929 22:50:45.341152 22585 raft_consensus.cc:3037] T f4e8fbb291994aa88233224f12f35a57 P a46aea30b8a8431189d809e841b82f6e [term 0 FOLLOWER]: Advancing to term 1
I20260929 22:50:45.342083 22585 raft_consensus.cc:2454] T f4e8fbb291994aa88233224f12f35a57 P a46aea30b8a8431189d809e841b82f6e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1144095f691a4cac85b7a4890ae1ae78 in term 1.
I20260929 22:50:45.342083 22729 raft_consensus.cc:2454] T f4e8fbb291994aa88233224f12f35a57 P 60d7fd130c0f46ea83489a52a291a07a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1144095f691a4cac85b7a4890ae1ae78 in term 1.
I20260929 22:50:45.342288 22803 leader_election.cc:302] T f4e8fbb291994aa88233224f12f35a57 P 1144095f691a4cac85b7a4890ae1ae78 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1144095f691a4cac85b7a4890ae1ae78, 60d7fd130c0f46ea83489a52a291a07a; no voters:
I20260929 22:50:45.342386 22952 raft_consensus.cc:2781] T f4e8fbb291994aa88233224f12f35a57 P 1144095f691a4cac85b7a4890ae1ae78 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 22:50:45.342561 22952 raft_consensus.cc:686] T f4e8fbb291994aa88233224f12f35a57 P 1144095f691a4cac85b7a4890ae1ae78 [term 1 LEADER]: Becoming Leader. State: Replica: 1144095f691a4cac85b7a4890ae1ae78, State: Running, Role: LEADER
I20260929 22:50:45.342654 22952 consensus_queue.cc:237] T f4e8fbb291994aa88233224f12f35a57 P 1144095f691a4cac85b7a4890ae1ae78 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a46aea30b8a8431189d809e841b82f6e" member_type: VOTER last_known_addr { host: "127.14.147.193" port: 33167 } } peers { permanent_uuid: "1144095f691a4cac85b7a4890ae1ae78" member_type: VOTER last_known_addr { host: "127.14.147.195" port: 35113 } } peers { permanent_uuid: "60d7fd130c0f46ea83489a52a291a07a" member_type: VOTER last_known_addr { host: "127.14.147.194" port: 44459 } }
I20260929 22:50:45.343541 22428 catalog_manager.cc:5495] T f4e8fbb291994aa88233224f12f35a57 P 1144095f691a4cac85b7a4890ae1ae78 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1144095f691a4cac85b7a4890ae1ae78 (127.14.147.195). New cstate: current_term: 1 leader_uuid: "1144095f691a4cac85b7a4890ae1ae78" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a46aea30b8a8431189d809e841b82f6e" member_type: VOTER last_known_addr { host: "127.14.147.193" port: 33167 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1144095f691a4cac85b7a4890ae1ae78" member_type: VOTER last_known_addr { host: "127.14.147.195" port: 35113 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "60d7fd130c0f46ea83489a52a291a07a" member_type: VOTER last_known_addr { host: "127.14.147.194" port: 44459 } health_report { overall_health: UNKNOWN } } }
W20260929 22:50:45.347368 22776 tablet.cc:2369] T f4e8fbb291994aa88233224f12f35a57 P 60d7fd130c0f46ea83489a52a291a07a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 22:50:45.359207 22941 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:50:45.356595 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49084 (local address 127.14.147.195:35113)
0929 22:50:45.356633 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0929 22:50:45.356635 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 22:50:45.356643 (+ 8us) server_negotiation.cc:407] Connection header received
0929 22:50:45.356699 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 22:50:45.356700 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 22:50:45.356717 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 22:50:45.356777 (+ 60us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 22:50:45.356887 (+ 110us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:50:45.357351 (+ 464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:50:45.358350 (+ 999us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:50:45.358565 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:50:45.358643 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 22:50:45.358864 (+ 221us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 22:50:45.358977 (+ 113us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 22:50:45.359031 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0929 22:50:45.359112 (+ 81us) server_negotiation.cc:299] Negotiation successful
0929 22:50:45.359135 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260929 22:50:45.361040 22729 raft_consensus.cc:1261] T f4e8fbb291994aa88233224f12f35a57 P 60d7fd130c0f46ea83489a52a291a07a [term 1 FOLLOWER]: Refusing update from remote peer 1144095f691a4cac85b7a4890ae1ae78: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 22:50:45.361073 22585 raft_consensus.cc:1261] T f4e8fbb291994aa88233224f12f35a57 P a46aea30b8a8431189d809e841b82f6e [term 1 FOLLOWER]: Refusing update from remote peer 1144095f691a4cac85b7a4890ae1ae78: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 22:50:45.361433 22954 consensus_queue.cc:1035] T f4e8fbb291994aa88233224f12f35a57 P 1144095f691a4cac85b7a4890ae1ae78 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a46aea30b8a8431189d809e841b82f6e" member_type: VOTER last_known_addr { host: "127.14.147.193" port: 33167 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 22:50:45.361570 22952 consensus_queue.cc:1035] T f4e8fbb291994aa88233224f12f35a57 P 1144095f691a4cac85b7a4890ae1ae78 [LEADER]: Connected to new peer: Peer: permanent_uuid: "60d7fd130c0f46ea83489a52a291a07a" member_type: VOTER last_known_addr { host: "127.14.147.194" port: 44459 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 22:50:45.364312 22957 mvcc.cc:205] Tried to move back new op lower bound from 7334798316996366336 to 7334798316924231680. Current Snapshot: MvccSnapshot[applied={T|T < 7334798316996366336}]
I20260929 22:50:45.366817 22430 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:46512:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 22:50:45.366914 22430 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:46512:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 22:50:45.367628 22430 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 7c99fb093ec947debdad2c01a7fd8027: Sending DeleteTablet for 3 replicas of tablet f4e8fbb291994aa88233224f12f35a57
I20260929 22:50:45.367939 22835 tablet_service.cc:1514] Processing DeleteTablet for tablet f4e8fbb291994aa88233224f12f35a57 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 22:50:45 UTC) from {username='oryx', principal='oryx/127.14.147.254@KRBTEST.COM'} at 127.0.0.1:49068
I20260929 22:50:45.367950 22709 tablet_service.cc:1514] Processing DeleteTablet for tablet f4e8fbb291994aa88233224f12f35a57 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 22:50:45 UTC) from {username='oryx', principal='oryx/127.14.147.254@KRBTEST.COM'} at 127.0.0.1:46792
I20260929 22:50:45.368237 22568 tablet_service.cc:1514] Processing DeleteTablet for tablet f4e8fbb291994aa88233224f12f35a57 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 22:50:45 UTC) from {username='oryx', principal='oryx/127.14.147.254@KRBTEST.COM'} at 127.0.0.1:41272
I20260929 22:50:45.368371 22430 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:46512:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260929 22:50:45.368496 22969 tablet_replica.cc:330] T f4e8fbb291994aa88233224f12f35a57 P a46aea30b8a8431189d809e841b82f6e: stopping tablet replica
W20260929 22:50:45.368512 22430 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 22:50:45.368597 22969 raft_consensus.cc:2229] T f4e8fbb291994aa88233224f12f35a57 P a46aea30b8a8431189d809e841b82f6e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 22:50:45.368721 22969 raft_consensus.cc:2258] T f4e8fbb291994aa88233224f12f35a57 P a46aea30b8a8431189d809e841b82f6e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 22:50:45.369266 22968 tablet_replica.cc:330] T f4e8fbb291994aa88233224f12f35a57 P 60d7fd130c0f46ea83489a52a291a07a: stopping tablet replica
I20260929 22:50:45.369616 22969 ts_tablet_manager.cc:1905] T f4e8fbb291994aa88233224f12f35a57 P a46aea30b8a8431189d809e841b82f6e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 22:50:45.369711 22835 tablet_service.cc:1467] Processing CreateTablet for tablet c72268163c6c48fe8a5ff982b28d7408 (DEFAULT_TABLE table=test-table [id=35d3e22dab834129a5bea999b2a686a9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 22:50:45.369853 22835 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c72268163c6c48fe8a5ff982b28d7408. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 22:50:45.369771 22709 tablet_service.cc:1467] Processing CreateTablet for tablet c72268163c6c48fe8a5ff982b28d7408 (DEFAULT_TABLE table=test-table [id=35d3e22dab834129a5bea999b2a686a9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 22:50:45.369773 22568 tablet_service.cc:1467] Processing CreateTablet for tablet c72268163c6c48fe8a5ff982b28d7408 (DEFAULT_TABLE table=test-table [id=35d3e22dab834129a5bea999b2a686a9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 22:50:45.369977 22709 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c72268163c6c48fe8a5ff982b28d7408. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 22:50:45.370054 22568 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c72268163c6c48fe8a5ff982b28d7408. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 22:50:45.371264 22944 tablet_bootstrap.cc:492] T c72268163c6c48fe8a5ff982b28d7408 P 1144095f691a4cac85b7a4890ae1ae78: Bootstrap starting.
I20260929 22:50:45.371260 22946 tablet_bootstrap.cc:492] T c72268163c6c48fe8a5ff982b28d7408 P 60d7fd130c0f46ea83489a52a291a07a: Bootstrap starting.
I20260929 22:50:45.371713 22944 tablet_bootstrap.cc:654] T c72268163c6c48fe8a5ff982b28d7408 P 1144095f691a4cac85b7a4890ae1ae78: Neither blocks nor log segments found. Creating new log.
I20260929 22:50:45.371721 22946 tablet_bootstrap.cc:654] T c72268163c6c48fe8a5ff982b28d7408 P 60d7fd130c0f46ea83489a52a291a07a: Neither blocks nor log segments found. Creating new log.
I20260929 22:50:45.371711 22945 tablet_bootstrap.cc:492] T c72268163c6c48fe8a5ff982b28d7408 P a46aea30b8a8431189d809e841b82f6e: Bootstrap starting.
I20260929 22:50:45.372069 22968 raft_consensus.cc:2229] T f4e8fbb291994aa88233224f12f35a57 P 60d7fd130c0f46ea83489a52a291a07a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 22:50:45.372115 22945 tablet_bootstrap.cc:654] T c72268163c6c48fe8a5ff982b28d7408 P a46aea30b8a8431189d809e841b82f6e: Neither blocks nor log segments found. Creating new log.
I20260929 22:50:45.372601 22944 tablet_bootstrap.cc:492] T c72268163c6c48fe8a5ff982b28d7408 P 1144095f691a4cac85b7a4890ae1ae78: No bootstrap required, opened a new log
I20260929 22:50:45.372646 22944 ts_tablet_manager.cc:1397] T c72268163c6c48fe8a5ff982b28d7408 P 1144095f691a4cac85b7a4890ae1ae78: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260929 22:50:45.372807 22944 raft_consensus.cc:348] T c72268163c6c48fe8a5ff982b28d7408 P 1144095f691a4cac85b7a4890ae1ae78 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a46aea30b8a8431189d809e841b82f6e" member_type: VOTER last_known_addr { host: "127.14.147.193" port: 33167 } } peers { permanent_uuid: "60d7fd130c0f46ea83489a52a291a07a" member_type: VOTER last_known_addr { host: "127.14.147.194" port: 44459 } } peers { permanent_uuid: "1144095f691a4cac85b7a4890ae1ae78" member_type: VOTER last_known_addr { host: "127.14.147.195" port: 35113 } }
I20260929 22:50:45.372867 22944 raft_consensus.cc:374] T c72268163c6c48fe8a5ff982b28d7408 P 1144095f691a4cac85b7a4890ae1ae78 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 22:50:45.372879 22944 raft_consensus.cc:729] T c72268163c6c48fe8a5ff982b28d7408 P 1144095f691a4cac85b7a4890ae1ae78 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1144095f691a4cac85b7a4890ae1ae78, State: Initialized, Role: FOLLOWER
I20260929 22:50:45.372916 22944 consensus_queue.cc:260] T c72268163c6c48fe8a5ff982b28d7408 P 1144095f691a4cac85b7a4890ae1ae78 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a46aea30b8a8431189d809e841b82f6e" member_type: VOTER last_known_addr { host: "127.14.147.193" port: 33167 } } peers { permanent_uuid: "60d7fd130c0f46ea83489a52a291a07a" member_type: VOTER last_known_addr { host: "127.14.147.194" port: 44459 } } peers { permanent_uuid: "1144095f691a4cac85b7a4890ae1ae78" member_type: VOTER last_known_addr { host: "127.14.147.195" port: 35113 } }
I20260929 22:50:45.372995 22944 ts_tablet_manager.cc:1428] T c72268163c6c48fe8a5ff982b28d7408 P 1144095f691a4cac85b7a4890ae1ae78: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260929 22:50:45.372738 22946 tablet_bootstrap.cc:492] T c72268163c6c48fe8a5ff982b28d7408 P 60d7fd130c0f46ea83489a52a291a07a: No bootstrap required, opened a new log
I20260929 22:50:45.373602 22946 ts_tablet_manager.cc:1397] T c72268163c6c48fe8a5ff982b28d7408 P 60d7fd130c0f46ea83489a52a291a07a: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260929 22:50:45.373690 22968 raft_consensus.cc:2258] T f4e8fbb291994aa88233224f12f35a57 P 60d7fd130c0f46ea83489a52a291a07a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 22:50:45.373757 22946 raft_consensus.cc:348] T c72268163c6c48fe8a5ff982b28d7408 P 60d7fd130c0f46ea83489a52a291a07a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a46aea30b8a8431189d809e841b82f6e" member_type: VOTER last_known_addr { host: "127.14.147.193" port: 33167 } } peers { permanent_uuid: "60d7fd130c0f46ea83489a52a291a07a" member_type: VOTER last_known_addr { host: "127.14.147.194" port: 44459 } } peers { permanent_uuid: "1144095f691a4cac85b7a4890ae1ae78" member_type: VOTER last_known_addr { host: "127.14.147.195" port: 35113 } }
I20260929 22:50:45.373854 22946 raft_consensus.cc:374] T c72268163c6c48fe8a5ff982b28d7408 P 60d7fd130c0f46ea83489a52a291a07a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 22:50:45.373888 22946 raft_consensus.cc:729] T c72268163c6c48fe8a5ff982b28d7408 P 60d7fd130c0f46ea83489a52a291a07a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 60d7fd130c0f46ea83489a52a291a07a, State: Initialized, Role: FOLLOWER
I20260929 22:50:45.373890 22967 tablet_replica.cc:330] T f4e8fbb291994aa88233224f12f35a57 P 1144095f691a4cac85b7a4890ae1ae78: stopping tablet replica
I20260929 22:50:45.373929 22946 consensus_queue.cc:260] T c72268163c6c48fe8a5ff982b28d7408 P 60d7fd130c0f46ea83489a52a291a07a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a46aea30b8a8431189d809e841b82f6e" member_type: VOTER last_known_addr { host: "127.14.147.193" port: 33167 } } peers { permanent_uuid: "60d7fd130c0f46ea83489a52a291a07a" member_type: VOTER last_known_addr { host: "127.14.147.194" port: 44459 } } peers { permanent_uuid: "1144095f691a4cac85b7a4890ae1ae78" member_type: VOTER last_known_addr { host: "127.14.147.195" port: 35113 } }
I20260929 22:50:45.373993 22967 raft_consensus.cc:2229] T f4e8fbb291994aa88233224f12f35a57 P 1144095f691a4cac85b7a4890ae1ae78 [term 1 LEADER]: Raft consensus shutting down.
I20260929 22:50:45.374025 22946 ts_tablet_manager.cc:1428] T c72268163c6c48fe8a5ff982b28d7408 P 60d7fd130c0f46ea83489a52a291a07a: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260929 22:50:45.374096 22967 raft_consensus.cc:2258] T f4e8fbb291994aa88233224f12f35a57 P 1144095f691a4cac85b7a4890ae1ae78 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 22:50:45.374382 22969 ts_tablet_manager.cc:1918] T f4e8fbb291994aa88233224f12f35a57 P a46aea30b8a8431189d809e841b82f6e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 22:50:45.374454 22969 log.cc:1196] T f4e8fbb291994aa88233224f12f35a57 P a46aea30b8a8431189d809e841b82f6e: Deleting WAL directory at /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790722213368738-14927-0/minicluster-data/ts-0/wal/wals/f4e8fbb291994aa88233224f12f35a57
I20260929 22:50:45.374553 22968 ts_tablet_manager.cc:1905] T f4e8fbb291994aa88233224f12f35a57 P 60d7fd130c0f46ea83489a52a291a07a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 22:50:45.374714 22969 ts_tablet_manager.cc:1939] T f4e8fbb291994aa88233224f12f35a57 P a46aea30b8a8431189d809e841b82f6e: Deleting consensus metadata
I20260929 22:50:45.375102 22417 catalog_manager.cc:4841] TS a46aea30b8a8431189d809e841b82f6e (127.14.147.193:33167): tablet f4e8fbb291994aa88233224f12f35a57 (table test-table [id=2dfa0c77190741728651d32c95aef52f]) successfully deleted
I20260929 22:50:45.375903 22967 ts_tablet_manager.cc:1905] T f4e8fbb291994aa88233224f12f35a57 P 1144095f691a4cac85b7a4890ae1ae78: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 22:50:45.376166 22945 tablet_bootstrap.cc:492] T c72268163c6c48fe8a5ff982b28d7408 P a46aea30b8a8431189d809e841b82f6e: No bootstrap required, opened a new log
I20260929 22:50:45.376224 22945 ts_tablet_manager.cc:1397] T c72268163c6c48fe8a5ff982b28d7408 P a46aea30b8a8431189d809e841b82f6e: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.000s
I20260929 22:50:45.376250 22968 ts_tablet_manager.cc:1918] T f4e8fbb291994aa88233224f12f35a57 P 60d7fd130c0f46ea83489a52a291a07a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 22:50:45.376330 22968 log.cc:1196] T f4e8fbb291994aa88233224f12f35a57 P 60d7fd130c0f46ea83489a52a291a07a: Deleting WAL directory at /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790722213368738-14927-0/minicluster-data/ts-1/wal/wals/f4e8fbb291994aa88233224f12f35a57
I20260929 22:50:45.376380 22945 raft_consensus.cc:348] T c72268163c6c48fe8a5ff982b28d7408 P a46aea30b8a8431189d809e841b82f6e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a46aea30b8a8431189d809e841b82f6e" member_type: VOTER last_known_addr { host: "127.14.147.193" port: 33167 } } peers { permanent_uuid: "60d7fd130c0f46ea83489a52a291a07a" member_type: VOTER last_known_addr { host: "127.14.147.194" port: 44459 } } peers { permanent_uuid: "1144095f691a4cac85b7a4890ae1ae78" member_type: VOTER last_known_addr { host: "127.14.147.195" port: 35113 } }
I20260929 22:50:45.376457 22945 raft_consensus.cc:374] T c72268163c6c48fe8a5ff982b28d7408 P a46aea30b8a8431189d809e841b82f6e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 22:50:45.376477 22945 raft_consensus.cc:729] T c72268163c6c48fe8a5ff982b28d7408 P a46aea30b8a8431189d809e841b82f6e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a46aea30b8a8431189d809e841b82f6e, State: Initialized, Role: FOLLOWER
I20260929 22:50:45.376525 22945 consensus_queue.cc:260] T c72268163c6c48fe8a5ff982b28d7408 P a46aea30b8a8431189d809e841b82f6e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a46aea30b8a8431189d809e841b82f6e" member_type: VOTER last_known_addr { host: "127.14.147.193" port: 33167 } } peers { permanent_uuid: "60d7fd130c0f46ea83489a52a291a07a" member_type: VOTER last_known_addr { host: "127.14.147.194" port: 44459 } } peers { permanent_uuid: "1144095f691a4cac85b7a4890ae1ae78" member_type: VOTER last_known_addr { host: "127.14.147.195" port: 35113 } }
I20260929 22:50:45.376579 22968 ts_tablet_manager.cc:1939] T f4e8fbb291994aa88233224f12f35a57 P 60d7fd130c0f46ea83489a52a291a07a: Deleting consensus metadata
I20260929 22:50:45.376619 22945 ts_tablet_manager.cc:1428] T c72268163c6c48fe8a5ff982b28d7408 P a46aea30b8a8431189d809e841b82f6e: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260929 22:50:45.376935 22417 catalog_manager.cc:4841] TS 60d7fd130c0f46ea83489a52a291a07a (127.14.147.194:44459): tablet f4e8fbb291994aa88233224f12f35a57 (table test-table [id=2dfa0c77190741728651d32c95aef52f]) successfully deleted
I20260929 22:50:45.377522 22967 ts_tablet_manager.cc:1918] T f4e8fbb291994aa88233224f12f35a57 P 1144095f691a4cac85b7a4890ae1ae78: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 22:50:45.377581 22967 log.cc:1196] T f4e8fbb291994aa88233224f12f35a57 P 1144095f691a4cac85b7a4890ae1ae78: Deleting WAL directory at /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790722213368738-14927-0/minicluster-data/ts-2/wal/wals/f4e8fbb291994aa88233224f12f35a57
I20260929 22:50:45.377774 22967 ts_tablet_manager.cc:1939] T f4e8fbb291994aa88233224f12f35a57 P 1144095f691a4cac85b7a4890ae1ae78: Deleting consensus metadata
I20260929 22:50:45.378139 22416 catalog_manager.cc:4841] TS 1144095f691a4cac85b7a4890ae1ae78 (127.14.147.195:35113): tablet f4e8fbb291994aa88233224f12f35a57 (table test-table [id=2dfa0c77190741728651d32c95aef52f]) successfully deleted
W20260929 22:50:45.439581 22635 tablet.cc:2369] T c72268163c6c48fe8a5ff982b28d7408 P a46aea30b8a8431189d809e841b82f6e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 22:50:45.486452 22951 raft_consensus.cc:482] T c72268163c6c48fe8a5ff982b28d7408 P 60d7fd130c0f46ea83489a52a291a07a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 22:50:45.486555 22951 raft_consensus.cc:504] T c72268163c6c48fe8a5ff982b28d7408 P 60d7fd130c0f46ea83489a52a291a07a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a46aea30b8a8431189d809e841b82f6e" member_type: VOTER last_known_addr { host: "127.14.147.193" port: 33167 } } peers { permanent_uuid: "60d7fd130c0f46ea83489a52a291a07a" member_type: VOTER last_known_addr { host: "127.14.147.194" port: 44459 } } peers { permanent_uuid: "1144095f691a4cac85b7a4890ae1ae78" member_type: VOTER last_known_addr { host: "127.14.147.195" port: 35113 } }
I20260929 22:50:45.486835 22951 leader_election.cc:288] T c72268163c6c48fe8a5ff982b28d7408 P 60d7fd130c0f46ea83489a52a291a07a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a46aea30b8a8431189d809e841b82f6e (127.14.147.193:33167), 1144095f691a4cac85b7a4890ae1ae78 (127.14.147.195:35113)
I20260929 22:50:45.490067 22940 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:50:45.487128 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.147.194:34831 (local address 127.14.147.193:33167)
0929 22:50:45.487173 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0929 22:50:45.487174 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0929 22:50:45.487186 (+ 12us) server_negotiation.cc:407] Connection header received
0929 22:50:45.487214 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 22:50:45.487215 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 22:50:45.487224 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 22:50:45.487273 (+ 49us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 22:50:45.487436 (+ 163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:50:45.487917 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:50:45.488898 (+ 981us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:50:45.489447 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:50:45.489523 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 22:50:45.489534 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0929 22:50:45.489945 (+ 411us) server_negotiation.cc:299] Negotiation successful
0929 22:50:45.489986 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260929 22:50:45.490067 22941 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:50:45.487108 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.147.194:54689 (local address 127.14.147.195:35113)
0929 22:50:45.487170 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0929 22:50:45.487171 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0929 22:50:45.487292 (+ 121us) server_negotiation.cc:407] Connection header received
0929 22:50:45.487337 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 22:50:45.487338 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 22:50:45.487349 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 22:50:45.487392 (+ 43us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 22:50:45.487575 (+ 183us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:50:45.488058 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:50:45.489037 (+ 979us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:50:45.489555 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:50:45.489616 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 22:50:45.489623 (+ 7us) server_negotiation.cc:1035] Waiting for connection context
0929 22:50:45.489942 (+ 319us) server_negotiation.cc:299] Negotiation successful
0929 22:50:45.489984 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260929 22:50:45.490217 22667 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:50:45.487026 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.147.193:33167 (local address 127.14.147.194:34831)
0929 22:50:45.487091 (+ 65us) negotiation.cc:107] Waiting for socket to connect
0929 22:50:45.487103 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0929 22:50:45.487150 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 22:50:45.487282 (+ 132us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 22:50:45.487283 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 22:50:45.487284 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 22:50:45.487356 (+ 72us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 22:50:45.487358 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 22:50:45.487999 (+ 641us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 22:50:45.488000 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 22:50:45.488806 (+ 806us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 22:50:45.488808 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 22:50:45.489664 (+ 856us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 22:50:45.489667 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 22:50:45.489832 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 22:50:45.489835 (+ 3us) client_negotiation.cc:769] Sending connection context
0929 22:50:45.489930 (+ 95us) client_negotiation.cc:240] Negotiation successful
0929 22:50:45.490078 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":44,"mutex_wait_us":62}
I20260929 22:50:45.490343 22973 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:50:45.487037 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.147.195:35113 (local address 127.14.147.194:54689)
0929 22:50:45.487214 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0929 22:50:45.487222 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0929 22:50:45.487280 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 22:50:45.487404 (+ 124us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 22:50:45.487405 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 22:50:45.487407 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 22:50:45.487511 (+ 104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 22:50:45.487512 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 22:50:45.488133 (+ 621us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 22:50:45.488134 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 22:50:45.488961 (+ 827us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 22:50:45.488964 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 22:50:45.489674 (+ 710us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 22:50:45.489675 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 22:50:45.489827 (+ 152us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 22:50:45.489836 (+ 9us) client_negotiation.cc:769] Sending connection context
0929 22:50:45.489930 (+ 94us) client_negotiation.cc:240] Negotiation successful
0929 22:50:45.490074 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":166,"mutex_wait_us":7,"thread_start_us":59,"threads_started":1}
I20260929 22:50:45.490576 22585 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c72268163c6c48fe8a5ff982b28d7408" candidate_uuid: "60d7fd130c0f46ea83489a52a291a07a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a46aea30b8a8431189d809e841b82f6e" is_pre_election: true
I20260929 22:50:45.490576 22870 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c72268163c6c48fe8a5ff982b28d7408" candidate_uuid: "60d7fd130c0f46ea83489a52a291a07a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1144095f691a4cac85b7a4890ae1ae78" is_pre_election: true
I20260929 22:50:45.490682 22585 raft_consensus.cc:2454] T c72268163c6c48fe8a5ff982b28d7408 P a46aea30b8a8431189d809e841b82f6e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60d7fd130c0f46ea83489a52a291a07a in term 0.
I20260929 22:50:45.490700 22870 raft_consensus.cc:2454] T c72268163c6c48fe8a5ff982b28d7408 P 1144095f691a4cac85b7a4890ae1ae78 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60d7fd130c0f46ea83489a52a291a07a in term 0.
I20260929 22:50:45.490886 22662 leader_election.cc:302] T c72268163c6c48fe8a5ff982b28d7408 P 60d7fd130c0f46ea83489a52a291a07a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d7fd130c0f46ea83489a52a291a07a, a46aea30b8a8431189d809e841b82f6e; no voters:
I20260929 22:50:45.491022 22951 raft_consensus.cc:2781] T c72268163c6c48fe8a5ff982b28d7408 P 60d7fd130c0f46ea83489a52a291a07a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 22:50:45.491058 22951 raft_consensus.cc:482] T c72268163c6c48fe8a5ff982b28d7408 P 60d7fd130c0f46ea83489a52a291a07a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 22:50:45.491086 22951 raft_consensus.cc:3037] T c72268163c6c48fe8a5ff982b28d7408 P 60d7fd130c0f46ea83489a52a291a07a [term 0 FOLLOWER]: Advancing to term 1
I20260929 22:50:45.491834 22951 raft_consensus.cc:504] T c72268163c6c48fe8a5ff982b28d7408 P 60d7fd130c0f46ea83489a52a291a07a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a46aea30b8a8431189d809e841b82f6e" member_type: VOTER last_known_addr { host: "127.14.147.193" port: 33167 } } peers { permanent_uuid: "60d7fd130c0f46ea83489a52a291a07a" member_type: VOTER last_known_addr { host: "127.14.147.194" port: 44459 } } peers { permanent_uuid: "1144095f691a4cac85b7a4890ae1ae78" member_type: VOTER last_known_addr { host: "127.14.147.195" port: 35113 } }
I20260929 22:50:45.491961 22951 leader_election.cc:288] T c72268163c6c48fe8a5ff982b28d7408 P 60d7fd130c0f46ea83489a52a291a07a [CANDIDATE]: Term 1 election: Requested vote from peers a46aea30b8a8431189d809e841b82f6e (127.14.147.193:33167), 1144095f691a4cac85b7a4890ae1ae78 (127.14.147.195:35113)
I20260929 22:50:45.492115 22585 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c72268163c6c48fe8a5ff982b28d7408" candidate_uuid: "60d7fd130c0f46ea83489a52a291a07a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a46aea30b8a8431189d809e841b82f6e"
I20260929 22:50:45.492177 22585 raft_consensus.cc:3037] T c72268163c6c48fe8a5ff982b28d7408 P a46aea30b8a8431189d809e841b82f6e [term 0 FOLLOWER]: Advancing to term 1
I20260929 22:50:45.492136 22870 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c72268163c6c48fe8a5ff982b28d7408" candidate_uuid: "60d7fd130c0f46ea83489a52a291a07a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1144095f691a4cac85b7a4890ae1ae78"
I20260929 22:50:45.492210 22870 raft_consensus.cc:3037] T c72268163c6c48fe8a5ff982b28d7408 P 1144095f691a4cac85b7a4890ae1ae78 [term 0 FOLLOWER]: Advancing to term 1
I20260929 22:50:45.492734 22585 raft_consensus.cc:2454] T c72268163c6c48fe8a5ff982b28d7408 P a46aea30b8a8431189d809e841b82f6e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 60d7fd130c0f46ea83489a52a291a07a in term 1.
I20260929 22:50:45.492906 22870 raft_consensus.cc:2454] T c72268163c6c48fe8a5ff982b28d7408 P 1144095f691a4cac85b7a4890ae1ae78 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 60d7fd130c0f46ea83489a52a291a07a in term 1.
I20260929 22:50:45.492905 22662 leader_election.cc:302] T c72268163c6c48fe8a5ff982b28d7408 P 60d7fd130c0f46ea83489a52a291a07a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d7fd130c0f46ea83489a52a291a07a, a46aea30b8a8431189d809e841b82f6e; no voters:
I20260929 22:50:45.493014 22951 raft_consensus.cc:2781] T c72268163c6c48fe8a5ff982b28d7408 P 60d7fd130c0f46ea83489a52a291a07a [term 1 FOLLOWER]: Leader election won for term 1
I20260929 22:50:45.493134 22951 raft_consensus.cc:686] T c72268163c6c48fe8a5ff982b28d7408 P 60d7fd130c0f46ea83489a52a291a07a [term 1 LEADER]: Becoming Leader. State: Replica: 60d7fd130c0f46ea83489a52a291a07a, State: Running, Role: LEADER
I20260929 22:50:45.493197 22951 consensus_queue.cc:237] T c72268163c6c48fe8a5ff982b28d7408 P 60d7fd130c0f46ea83489a52a291a07a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a46aea30b8a8431189d809e841b82f6e" member_type: VOTER last_known_addr { host: "127.14.147.193" port: 33167 } } peers { permanent_uuid: "60d7fd130c0f46ea83489a52a291a07a" member_type: VOTER last_known_addr { host: "127.14.147.194" port: 44459 } } peers { permanent_uuid: "1144095f691a4cac85b7a4890ae1ae78" member_type: VOTER last_known_addr { host: "127.14.147.195" port: 35113 } }
I20260929 22:50:45.493893 22430 catalog_manager.cc:5495] T c72268163c6c48fe8a5ff982b28d7408 P 60d7fd130c0f46ea83489a52a291a07a reported cstate change: term changed from 0 to 1, leader changed from <none> to 60d7fd130c0f46ea83489a52a291a07a (127.14.147.194). New cstate: current_term: 1 leader_uuid: "60d7fd130c0f46ea83489a52a291a07a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a46aea30b8a8431189d809e841b82f6e" member_type: VOTER last_known_addr { host: "127.14.147.193" port: 33167 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "60d7fd130c0f46ea83489a52a291a07a" member_type: VOTER last_known_addr { host: "127.14.147.194" port: 44459 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1144095f691a4cac85b7a4890ae1ae78" member_type: VOTER last_known_addr { host: "127.14.147.195" port: 35113 } health_report { overall_health: UNKNOWN } } }
W20260929 22:50:45.495745 22917 tablet.cc:2369] T c72268163c6c48fe8a5ff982b28d7408 P 1144095f691a4cac85b7a4890ae1ae78: Can't schedule compaction. Clean time has not been advanced past its initial value.
Sep 29 22:50:45 dist-test-slave-98mk krb5kdc[22385](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790722244, etypes {rep=17 tkt=17 ses=17}, oryx/127.14.147.254@KRBTEST.COM for oryx/127.14.147.254@KRBTEST.COM
I20260929 22:50:45.547560 22480 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:50:45.540261 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46520 (local address 127.14.147.254:33005)
0929 22:50:45.540332 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0929 22:50:45.540334 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 22:50:45.540534 (+ 200us) server_negotiation.cc:407] Connection header received
0929 22:50:45.540592 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 22:50:45.540594 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 22:50:45.540612 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 22:50:45.540718 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0929 22:50:45.541226 (+ 508us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:50:45.541762 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:50:45.542613 (+ 851us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:50:45.542845 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:50:45.542932 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 22:50:45.545121 (+ 2189us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 22:50:45.545123 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 22:50:45.545123 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 22:50:45.545130 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0929 22:50:45.546566 (+ 1436us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 22:50:45.546935 (+ 369us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 22:50:45.546936 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 22:50:45.546936 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 22:50:45.546982 (+ 46us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 22:50:45.547113 (+ 131us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 22:50:45.547113 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 22:50:45.547114 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 22:50:45.547215 (+ 101us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 22:50:45.547258 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0929 22:50:45.547367 (+ 109us) server_negotiation.cc:299] Negotiation successful
0929 22:50:45.547487 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260929 22:50:45.547677 22981 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:50:45.540137 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.147.254:33005 (local address 127.0.0.1:46520)
0929 22:50:45.540347 (+ 210us) negotiation.cc:107] Waiting for socket to connect
0929 22:50:45.540357 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0929 22:50:45.540448 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 22:50:45.540734 (+ 286us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 22:50:45.540735 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 22:50:45.541061 (+ 326us) client_negotiation.cc:189] Negotiated authn=SASL
0929 22:50:45.541121 (+ 60us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 22:50:45.541125 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 22:50:45.541842 (+ 717us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 22:50:45.541843 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 22:50:45.542459 (+ 616us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 22:50:45.542470 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 22:50:45.542952 (+ 482us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 22:50:45.542954 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 22:50:45.543011 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 22:50:45.543634 (+ 623us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 22:50:45.543639 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0929 22:50:45.545024 (+ 1385us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 22:50:45.546651 (+ 1627us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 22:50:45.546660 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 22:50:45.546662 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 22:50:45.546877 (+ 215us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 22:50:45.547026 (+ 149us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 22:50:45.547026 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 22:50:45.547027 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 22:50:45.547075 (+ 48us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 22:50:45.547255 (+ 180us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 22:50:45.547259 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 22:50:45.547300 (+ 41us) client_negotiation.cc:769] Sending connection context
0929 22:50:45.547345 (+ 45us) client_negotiation.cc:240] Negotiation successful
0929 22:50:45.547545 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":170,"mutex_wait_us":105,"thread_start_us":92,"threads_started":1}
I20260929 22:50:45.549526 22430 catalog_manager.cc:1908] Servicing CreateTable request from {username='oryx', principal='oryx/127.14.147.254@KRBTEST.COM'} at 127.0.0.1:46520:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260929 22:50:45.549727 22430 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 22:50:45.551402 22709 tablet_service.cc:1467] Processing CreateTablet for tablet 925fab2279cf420b983720a7da5b8421 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=596156eb845144b3b48a8cb900b8e483]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260929 22:50:45.551389 22568 tablet_service.cc:1467] Processing CreateTablet for tablet 925fab2279cf420b983720a7da5b8421 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=596156eb845144b3b48a8cb900b8e483]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260929 22:50:45.551579 22709 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 925fab2279cf420b983720a7da5b8421. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 22:50:45.551580 22568 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 925fab2279cf420b983720a7da5b8421. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 22:50:45.552819 22946 tablet_bootstrap.cc:492] T 925fab2279cf420b983720a7da5b8421 P 60d7fd130c0f46ea83489a52a291a07a: Bootstrap starting.
I20260929 22:50:45.552903 22835 tablet_service.cc:1467] Processing CreateTablet for tablet 925fab2279cf420b983720a7da5b8421 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=596156eb845144b3b48a8cb900b8e483]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260929 22:50:45.553048 22835 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 925fab2279cf420b983720a7da5b8421. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 22:50:45.553226 22946 tablet_bootstrap.cc:654] T 925fab2279cf420b983720a7da5b8421 P 60d7fd130c0f46ea83489a52a291a07a: Neither blocks nor log segments found. Creating new log.
I20260929 22:50:45.553822 22946 tablet_bootstrap.cc:492] T 925fab2279cf420b983720a7da5b8421 P 60d7fd130c0f46ea83489a52a291a07a: No bootstrap required, opened a new log
I20260929 22:50:45.553864 22946 ts_tablet_manager.cc:1397] T 925fab2279cf420b983720a7da5b8421 P 60d7fd130c0f46ea83489a52a291a07a: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260929 22:50:45.553987 22946 raft_consensus.cc:348] T 925fab2279cf420b983720a7da5b8421 P 60d7fd130c0f46ea83489a52a291a07a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a46aea30b8a8431189d809e841b82f6e" member_type: VOTER last_known_addr { host: "127.14.147.193" port: 33167 } } peers { permanent_uuid: "60d7fd130c0f46ea83489a52a291a07a" member_type: VOTER last_known_addr { host: "127.14.147.194" port: 44459 } } peers { permanent_uuid: "1144095f691a4cac85b7a4890ae1ae78" member_type: VOTER last_known_addr { host: "127.14.147.195" port: 35113 } }
I20260929 22:50:45.554056 22946 raft_consensus.cc:374] T 925fab2279cf420b983720a7da5b8421 P 60d7fd130c0f46ea83489a52a291a07a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 22:50:45.554069 22946 raft_consensus.cc:729] T 925fab2279cf420b983720a7da5b8421 P 60d7fd130c0f46ea83489a52a291a07a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 60d7fd130c0f46ea83489a52a291a07a, State: Initialized, Role: FOLLOWER
I20260929 22:50:45.554104 22946 consensus_queue.cc:260] T 925fab2279cf420b983720a7da5b8421 P 60d7fd130c0f46ea83489a52a291a07a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a46aea30b8a8431189d809e841b82f6e" member_type: VOTER last_known_addr { host: "127.14.147.193" port: 33167 } } peers { permanent_uuid: "60d7fd130c0f46ea83489a52a291a07a" member_type: VOTER last_known_addr { host: "127.14.147.194" port: 44459 } } peers { permanent_uuid: "1144095f691a4cac85b7a4890ae1ae78" member_type: VOTER last_known_addr { host: "127.14.147.195" port: 35113 } }
I20260929 22:50:45.554196 22946 ts_tablet_manager.cc:1428] T 925fab2279cf420b983720a7da5b8421 P 60d7fd130c0f46ea83489a52a291a07a: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260929 22:50:45.554229 22951 tablet_replica.cc:439] T 925fab2279cf420b983720a7da5b8421 P 60d7fd130c0f46ea83489a52a291a07a: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a46aea30b8a8431189d809e841b82f6e" member_type: VOTER last_known_addr { host: "127.14.147.193" port: 33167 } } peers { permanent_uuid: "60d7fd130c0f46ea83489a52a291a07a" member_type: VOTER last_known_addr { host: "127.14.147.194" port: 44459 } } peers { permanent_uuid: "1144095f691a4cac85b7a4890ae1ae78" member_type: VOTER last_known_addr { host: "127.14.147.195" port: 35113 } } }
I20260929 22:50:45.554301 22951 tablet_replica.cc:442] T 925fab2279cf420b983720a7da5b8421 P 60d7fd130c0f46ea83489a52a291a07a: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 22:50:45.554301 22944 tablet_bootstrap.cc:492] T 925fab2279cf420b983720a7da5b8421 P 1144095f691a4cac85b7a4890ae1ae78: Bootstrap starting.
I20260929 22:50:45.554342 22945 tablet_bootstrap.cc:492] T 925fab2279cf420b983720a7da5b8421 P a46aea30b8a8431189d809e841b82f6e: Bootstrap starting.
I20260929 22:50:45.554639 22944 tablet_bootstrap.cc:654] T 925fab2279cf420b983720a7da5b8421 P 1144095f691a4cac85b7a4890ae1ae78: Neither blocks nor log segments found. Creating new log.
I20260929 22:50:45.554706 22945 tablet_bootstrap.cc:654] T 925fab2279cf420b983720a7da5b8421 P a46aea30b8a8431189d809e841b82f6e: Neither blocks nor log segments found. Creating new log.
I20260929 22:50:45.555384 22945 tablet_bootstrap.cc:492] T 925fab2279cf420b983720a7da5b8421 P a46aea30b8a8431189d809e841b82f6e: No bootstrap required, opened a new log
I20260929 22:50:45.555467 22945 ts_tablet_manager.cc:1397] T 925fab2279cf420b983720a7da5b8421 P a46aea30b8a8431189d809e841b82f6e: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260929 22:50:45.555612 22945 raft_consensus.cc:348] T 925fab2279cf420b983720a7da5b8421 P a46aea30b8a8431189d809e841b82f6e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a46aea30b8a8431189d809e841b82f6e" member_type: VOTER last_known_addr { host: "127.14.147.193" port: 33167 } } peers { permanent_uuid: "60d7fd130c0f46ea83489a52a291a07a" member_type: VOTER last_known_addr { host: "127.14.147.194" port: 44459 } } peers { permanent_uuid: "1144095f691a4cac85b7a4890ae1ae78" member_type: VOTER last_known_addr { host: "127.14.147.195" port: 35113 } }
I20260929 22:50:45.555675 22945 raft_consensus.cc:374] T 925fab2279cf420b983720a7da5b8421 P a46aea30b8a8431189d809e841b82f6e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 22:50:45.555691 22945 raft_consensus.cc:729] T 925fab2279cf420b983720a7da5b8421 P a46aea30b8a8431189d809e841b82f6e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a46aea30b8a8431189d809e841b82f6e, State: Initialized, Role: FOLLOWER
I20260929 22:50:45.555694 22944 tablet_bootstrap.cc:492] T 925fab2279cf420b983720a7da5b8421 P 1144095f691a4cac85b7a4890ae1ae78: No bootstrap required, opened a new log
I20260929 22:50:45.555745 22944 ts_tablet_manager.cc:1397] T 925fab2279cf420b983720a7da5b8421 P 1144095f691a4cac85b7a4890ae1ae78: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260929 22:50:45.555733 22945 consensus_queue.cc:260] T 925fab2279cf420b983720a7da5b8421 P a46aea30b8a8431189d809e841b82f6e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a46aea30b8a8431189d809e841b82f6e" member_type: VOTER last_known_addr { host: "127.14.147.193" port: 33167 } } peers { permanent_uuid: "60d7fd130c0f46ea83489a52a291a07a" member_type: VOTER last_known_addr { host: "127.14.147.194" port: 44459 } } peers { permanent_uuid: "1144095f691a4cac85b7a4890ae1ae78" member_type: VOTER last_known_addr { host: "127.14.147.195" port: 35113 } }
I20260929 22:50:45.555837 22945 ts_tablet_manager.cc:1428] T 925fab2279cf420b983720a7da5b8421 P a46aea30b8a8431189d809e841b82f6e: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260929 22:50:45.555868 22944 raft_consensus.cc:348] T 925fab2279cf420b983720a7da5b8421 P 1144095f691a4cac85b7a4890ae1ae78 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a46aea30b8a8431189d809e841b82f6e" member_type: VOTER last_known_addr { host: "127.14.147.193" port: 33167 } } peers { permanent_uuid: "60d7fd130c0f46ea83489a52a291a07a" member_type: VOTER last_known_addr { host: "127.14.147.194" port: 44459 } } peers { permanent_uuid: "1144095f691a4cac85b7a4890ae1ae78" member_type: VOTER last_known_addr { host: "127.14.147.195" port: 35113 } }
I20260929 22:50:45.555950 22944 raft_consensus.cc:374] T 925fab2279cf420b983720a7da5b8421 P 1144095f691a4cac85b7a4890ae1ae78 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 22:50:45.555975 22944 raft_consensus.cc:729] T 925fab2279cf420b983720a7da5b8421 P 1144095f691a4cac85b7a4890ae1ae78 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1144095f691a4cac85b7a4890ae1ae78, State: Initialized, Role: FOLLOWER
I20260929 22:50:45.556020 22944 consensus_queue.cc:260] T 925fab2279cf420b983720a7da5b8421 P 1144095f691a4cac85b7a4890ae1ae78 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a46aea30b8a8431189d809e841b82f6e" member_type: VOTER last_known_addr { host: "127.14.147.193" port: 33167 } } peers { permanent_uuid: "60d7fd130c0f46ea83489a52a291a07a" member_type: VOTER last_known_addr { host: "127.14.147.194" port: 44459 } } peers { permanent_uuid: "1144095f691a4cac85b7a4890ae1ae78" member_type: VOTER last_known_addr { host: "127.14.147.195" port: 35113 } }
I20260929 22:50:45.556100 22944 ts_tablet_manager.cc:1428] T 925fab2279cf420b983720a7da5b8421 P 1144095f691a4cac85b7a4890ae1ae78: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260929 22:50:45.556245 22950 tablet_replica.cc:439] T 925fab2279cf420b983720a7da5b8421 P a46aea30b8a8431189d809e841b82f6e: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a46aea30b8a8431189d809e841b82f6e" member_type: VOTER last_known_addr { host: "127.14.147.193" port: 33167 } } peers { permanent_uuid: "60d7fd130c0f46ea83489a52a291a07a" member_type: VOTER last_known_addr { host: "127.14.147.194" port: 44459 } } peers { permanent_uuid: "1144095f691a4cac85b7a4890ae1ae78" member_type: VOTER last_known_addr { host: "127.14.147.195" port: 35113 } } }
I20260929 22:50:45.556334 22950 tablet_replica.cc:442] T 925fab2279cf420b983720a7da5b8421 P a46aea30b8a8431189d809e841b82f6e: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 22:50:45.556427 22952 tablet_replica.cc:439] T 925fab2279cf420b983720a7da5b8421 P 1144095f691a4cac85b7a4890ae1ae78: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a46aea30b8a8431189d809e841b82f6e" member_type: VOTER last_known_addr { host: "127.14.147.193" port: 33167 } } peers { permanent_uuid: "60d7fd130c0f46ea83489a52a291a07a" member_type: VOTER last_known_addr { host: "127.14.147.194" port: 44459 } } peers { permanent_uuid: "1144095f691a4cac85b7a4890ae1ae78" member_type: VOTER last_known_addr { host: "127.14.147.195" port: 35113 } } }
I20260929 22:50:45.556495 22952 tablet_replica.cc:442] T 925fab2279cf420b983720a7da5b8421 P 1144095f691a4cac85b7a4890ae1ae78: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 22:50:45.563095 22952 raft_consensus.cc:482] T 925fab2279cf420b983720a7da5b8421 P 1144095f691a4cac85b7a4890ae1ae78 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 22:50:45.563198 22952 raft_consensus.cc:504] T 925fab2279cf420b983720a7da5b8421 P 1144095f691a4cac85b7a4890ae1ae78 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a46aea30b8a8431189d809e841b82f6e" member_type: VOTER last_known_addr { host: "127.14.147.193" port: 33167 } } peers { permanent_uuid: "60d7fd130c0f46ea83489a52a291a07a" member_type: VOTER last_known_addr { host: "127.14.147.194" port: 44459 } } peers { permanent_uuid: "1144095f691a4cac85b7a4890ae1ae78" member_type: VOTER last_known_addr { host: "127.14.147.195" port: 35113 } }
I20260929 22:50:45.563338 22952 leader_election.cc:288] T 925fab2279cf420b983720a7da5b8421 P 1144095f691a4cac85b7a4890ae1ae78 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a46aea30b8a8431189d809e841b82f6e (127.14.147.193:33167), 60d7fd130c0f46ea83489a52a291a07a (127.14.147.194:44459)
I20260929 22:50:45.563565 22585 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "925fab2279cf420b983720a7da5b8421" candidate_uuid: "1144095f691a4cac85b7a4890ae1ae78" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a46aea30b8a8431189d809e841b82f6e" is_pre_election: true
I20260929 22:50:45.563622 22729 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "925fab2279cf420b983720a7da5b8421" candidate_uuid: "1144095f691a4cac85b7a4890ae1ae78" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "60d7fd130c0f46ea83489a52a291a07a" is_pre_election: true
I20260929 22:50:45.563687 22585 raft_consensus.cc:2454] T 925fab2279cf420b983720a7da5b8421 P a46aea30b8a8431189d809e841b82f6e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1144095f691a4cac85b7a4890ae1ae78 in term 0.
I20260929 22:50:45.563712 22729 raft_consensus.cc:2454] T 925fab2279cf420b983720a7da5b8421 P 60d7fd130c0f46ea83489a52a291a07a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1144095f691a4cac85b7a4890ae1ae78 in term 0.
I20260929 22:50:45.563910 22803 leader_election.cc:302] T 925fab2279cf420b983720a7da5b8421 P 1144095f691a4cac85b7a4890ae1ae78 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1144095f691a4cac85b7a4890ae1ae78, a46aea30b8a8431189d809e841b82f6e; no voters:
I20260929 22:50:45.564038 22952 raft_consensus.cc:2781] T 925fab2279cf420b983720a7da5b8421 P 1144095f691a4cac85b7a4890ae1ae78 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 22:50:45.564091 22952 raft_consensus.cc:482] T 925fab2279cf420b983720a7da5b8421 P 1144095f691a4cac85b7a4890ae1ae78 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 22:50:45.564114 22952 raft_consensus.cc:3037] T 925fab2279cf420b983720a7da5b8421 P 1144095f691a4cac85b7a4890ae1ae78 [term 0 FOLLOWER]: Advancing to term 1
I20260929 22:50:45.564798 22952 raft_consensus.cc:504] T 925fab2279cf420b983720a7da5b8421 P 1144095f691a4cac85b7a4890ae1ae78 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a46aea30b8a8431189d809e841b82f6e" member_type: VOTER last_known_addr { host: "127.14.147.193" port: 33167 } } peers { permanent_uuid: "60d7fd130c0f46ea83489a52a291a07a" member_type: VOTER last_known_addr { host: "127.14.147.194" port: 44459 } } peers { permanent_uuid: "1144095f691a4cac85b7a4890ae1ae78" member_type: VOTER last_known_addr { host: "127.14.147.195" port: 35113 } }
I20260929 22:50:45.564911 22952 leader_election.cc:288] T 925fab2279cf420b983720a7da5b8421 P 1144095f691a4cac85b7a4890ae1ae78 [CANDIDATE]: Term 1 election: Requested vote from peers a46aea30b8a8431189d809e841b82f6e (127.14.147.193:33167), 60d7fd130c0f46ea83489a52a291a07a (127.14.147.194:44459)
I20260929 22:50:45.565085 22585 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "925fab2279cf420b983720a7da5b8421" candidate_uuid: "1144095f691a4cac85b7a4890ae1ae78" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a46aea30b8a8431189d809e841b82f6e"
I20260929 22:50:45.565177 22585 raft_consensus.cc:3037] T 925fab2279cf420b983720a7da5b8421 P a46aea30b8a8431189d809e841b82f6e [term 0 FOLLOWER]: Advancing to term 1
I20260929 22:50:45.565136 22729 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "925fab2279cf420b983720a7da5b8421" candidate_uuid: "1144095f691a4cac85b7a4890ae1ae78" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "60d7fd130c0f46ea83489a52a291a07a"
I20260929 22:50:45.565208 22729 raft_consensus.cc:3037] T 925fab2279cf420b983720a7da5b8421 P 60d7fd130c0f46ea83489a52a291a07a [term 0 FOLLOWER]: Advancing to term 1
I20260929 22:50:45.565762 22585 raft_consensus.cc:2454] T 925fab2279cf420b983720a7da5b8421 P a46aea30b8a8431189d809e841b82f6e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1144095f691a4cac85b7a4890ae1ae78 in term 1.
I20260929 22:50:45.565922 22803 leader_election.cc:302] T 925fab2279cf420b983720a7da5b8421 P 1144095f691a4cac85b7a4890ae1ae78 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1144095f691a4cac85b7a4890ae1ae78, a46aea30b8a8431189d809e841b82f6e; no voters:
I20260929 22:50:45.566027 22952 raft_consensus.cc:2781] T 925fab2279cf420b983720a7da5b8421 P 1144095f691a4cac85b7a4890ae1ae78 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 22:50:45.566087 22952 raft_consensus.cc:686] T 925fab2279cf420b983720a7da5b8421 P 1144095f691a4cac85b7a4890ae1ae78 [term 1 LEADER]: Becoming Leader. State: Replica: 1144095f691a4cac85b7a4890ae1ae78, State: Running, Role: LEADER
I20260929 22:50:45.566143 22952 consensus_queue.cc:237] T 925fab2279cf420b983720a7da5b8421 P 1144095f691a4cac85b7a4890ae1ae78 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a46aea30b8a8431189d809e841b82f6e" member_type: VOTER last_known_addr { host: "127.14.147.193" port: 33167 } } peers { permanent_uuid: "60d7fd130c0f46ea83489a52a291a07a" member_type: VOTER last_known_addr { host: "127.14.147.194" port: 44459 } } peers { permanent_uuid: "1144095f691a4cac85b7a4890ae1ae78" member_type: VOTER last_known_addr { host: "127.14.147.195" port: 35113 } }
I20260929 22:50:45.566205 22729 raft_consensus.cc:2454] T 925fab2279cf420b983720a7da5b8421 P 60d7fd130c0f46ea83489a52a291a07a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1144095f691a4cac85b7a4890ae1ae78 in term 1.
I20260929 22:50:45.566421 22954 tablet_replica.cc:439] T 925fab2279cf420b983720a7da5b8421 P 1144095f691a4cac85b7a4890ae1ae78: TxnStatusTablet state changed. Reason: New leader 1144095f691a4cac85b7a4890ae1ae78. Latest consensus state: current_term: 1 leader_uuid: "1144095f691a4cac85b7a4890ae1ae78" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a46aea30b8a8431189d809e841b82f6e" member_type: VOTER last_known_addr { host: "127.14.147.193" port: 33167 } } peers { permanent_uuid: "60d7fd130c0f46ea83489a52a291a07a" member_type: VOTER last_known_addr { host: "127.14.147.194" port: 44459 } } peers { permanent_uuid: "1144095f691a4cac85b7a4890ae1ae78" member_type: VOTER last_known_addr { host: "127.14.147.195" port: 35113 } } }
I20260929 22:50:45.566504 22954 tablet_replica.cc:442] T 925fab2279cf420b983720a7da5b8421 P 1144095f691a4cac85b7a4890ae1ae78: This TxnStatusTablet replica's current role is: LEADER
I20260929 22:50:45.566730 22428 catalog_manager.cc:5495] T 925fab2279cf420b983720a7da5b8421 P 1144095f691a4cac85b7a4890ae1ae78 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1144095f691a4cac85b7a4890ae1ae78 (127.14.147.195). New cstate: current_term: 1 leader_uuid: "1144095f691a4cac85b7a4890ae1ae78" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a46aea30b8a8431189d809e841b82f6e" member_type: VOTER last_known_addr { host: "127.14.147.193" port: 33167 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "60d7fd130c0f46ea83489a52a291a07a" member_type: VOTER last_known_addr { host: "127.14.147.194" port: 44459 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1144095f691a4cac85b7a4890ae1ae78" member_type: VOTER last_known_addr { host: "127.14.147.195" port: 35113 } health_report { overall_health: HEALTHY } } }
I20260929 22:50:45.566991 22986 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260929 22:50:45.567094 22986 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260929 22:50:45.572616 22981 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:50:45.570372 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.147.195:35113 (local address 127.0.0.1:49098)
0929 22:50:45.570435 (+ 63us) negotiation.cc:107] Waiting for socket to connect
0929 22:50:45.570447 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0929 22:50:45.570493 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 22:50:45.570620 (+ 127us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 22:50:45.570621 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 22:50:45.570622 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0929 22:50:45.570669 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 22:50:45.570672 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 22:50:45.571255 (+ 583us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 22:50:45.571256 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 22:50:45.571878 (+ 622us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 22:50:45.571881 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 22:50:45.572219 (+ 338us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 22:50:45.572220 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 22:50:45.572282 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 22:50:45.572297 (+ 15us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0929 22:50:45.572449 (+ 152us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0929 22:50:45.572456 (+ 7us) client_negotiation.cc:769] Sending connection context
0929 22:50:45.572496 (+ 40us) client_negotiation.cc:240] Negotiation successful
0929 22:50:45.572544 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":26}
I20260929 22:50:45.572616 22941 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:50:45.570431 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49098 (local address 127.14.147.195:35113)
0929 22:50:45.570486 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0929 22:50:45.570488 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 22:50:45.570497 (+ 9us) server_negotiation.cc:407] Connection header received
0929 22:50:45.570553 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 22:50:45.570554 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 22:50:45.570564 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 22:50:45.570614 (+ 50us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 22:50:45.570725 (+ 111us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:50:45.571190 (+ 465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:50:45.571955 (+ 765us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:50:45.572154 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:50:45.572208 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 22:50:45.572365 (+ 157us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 22:50:45.572400 (+ 35us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 22:50:45.572444 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0929 22:50:45.572498 (+ 54us) server_negotiation.cc:299] Negotiation successful
0929 22:50:45.572522 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260929 22:50:45.573582 22585 raft_consensus.cc:1261] T 925fab2279cf420b983720a7da5b8421 P a46aea30b8a8431189d809e841b82f6e [term 1 FOLLOWER]: Refusing update from remote peer 1144095f691a4cac85b7a4890ae1ae78: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 22:50:45.573585 22729 raft_consensus.cc:1261] T 925fab2279cf420b983720a7da5b8421 P 60d7fd130c0f46ea83489a52a291a07a [term 1 FOLLOWER]: Refusing update from remote peer 1144095f691a4cac85b7a4890ae1ae78: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 22:50:45.573786 22952 consensus_queue.cc:1035] T 925fab2279cf420b983720a7da5b8421 P 1144095f691a4cac85b7a4890ae1ae78 [LEADER]: Connected to new peer: Peer: permanent_uuid: "60d7fd130c0f46ea83489a52a291a07a" member_type: VOTER last_known_addr { host: "127.14.147.194" port: 44459 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 22:50:45.573904 22954 consensus_queue.cc:1035] T 925fab2279cf420b983720a7da5b8421 P 1144095f691a4cac85b7a4890ae1ae78 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a46aea30b8a8431189d809e841b82f6e" member_type: VOTER last_known_addr { host: "127.14.147.193" port: 33167 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 22:50:45.574301 22951 tablet_replica.cc:439] T 925fab2279cf420b983720a7da5b8421 P 60d7fd130c0f46ea83489a52a291a07a: TxnStatusTablet state changed. Reason: New leader 1144095f691a4cac85b7a4890ae1ae78. Latest consensus state: current_term: 1 leader_uuid: "1144095f691a4cac85b7a4890ae1ae78" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a46aea30b8a8431189d809e841b82f6e" member_type: VOTER last_known_addr { host: "127.14.147.193" port: 33167 } } peers { permanent_uuid: "60d7fd130c0f46ea83489a52a291a07a" member_type: VOTER last_known_addr { host: "127.14.147.194" port: 44459 } } peers { permanent_uuid: "1144095f691a4cac85b7a4890ae1ae78" member_type: VOTER last_known_addr { host: "127.14.147.195" port: 35113 } } }
I20260929 22:50:45.574383 22951 tablet_replica.cc:442] T 925fab2279cf420b983720a7da5b8421 P 60d7fd130c0f46ea83489a52a291a07a: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 22:50:45.574792 22950 tablet_replica.cc:439] T 925fab2279cf420b983720a7da5b8421 P a46aea30b8a8431189d809e841b82f6e: TxnStatusTablet state changed. Reason: New leader 1144095f691a4cac85b7a4890ae1ae78. Latest consensus state: current_term: 1 leader_uuid: "1144095f691a4cac85b7a4890ae1ae78" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a46aea30b8a8431189d809e841b82f6e" member_type: VOTER last_known_addr { host: "127.14.147.193" port: 33167 } } peers { permanent_uuid: "60d7fd130c0f46ea83489a52a291a07a" member_type: VOTER last_known_addr { host: "127.14.147.194" port: 44459 } } peers { permanent_uuid: "1144095f691a4cac85b7a4890ae1ae78" member_type: VOTER last_known_addr { host: "127.14.147.195" port: 35113 } } }
I20260929 22:50:45.574819 22954 tablet_replica.cc:439] T 925fab2279cf420b983720a7da5b8421 P 1144095f691a4cac85b7a4890ae1ae78: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "1144095f691a4cac85b7a4890ae1ae78" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a46aea30b8a8431189d809e841b82f6e" member_type: VOTER last_known_addr { host: "127.14.147.193" port: 33167 } } peers { permanent_uuid: "60d7fd130c0f46ea83489a52a291a07a" member_type: VOTER last_known_addr { host: "127.14.147.194" port: 44459 } } peers { permanent_uuid: "1144095f691a4cac85b7a4890ae1ae78" member_type: VOTER last_known_addr { host: "127.14.147.195" port: 35113 } } }
I20260929 22:50:45.574882 22950 tablet_replica.cc:442] T 925fab2279cf420b983720a7da5b8421 P a46aea30b8a8431189d809e841b82f6e: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 22:50:45.574888 22954 tablet_replica.cc:442] T 925fab2279cf420b983720a7da5b8421 P 1144095f691a4cac85b7a4890ae1ae78: This TxnStatusTablet replica's current role is: LEADER
I20260929 22:50:45.575719 22951 tablet_replica.cc:439] T 925fab2279cf420b983720a7da5b8421 P 60d7fd130c0f46ea83489a52a291a07a: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "1144095f691a4cac85b7a4890ae1ae78" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a46aea30b8a8431189d809e841b82f6e" member_type: VOTER last_known_addr { host: "127.14.147.193" port: 33167 } } peers { permanent_uuid: "60d7fd130c0f46ea83489a52a291a07a" member_type: VOTER last_known_addr { host: "127.14.147.194" port: 44459 } } peers { permanent_uuid: "1144095f691a4cac85b7a4890ae1ae78" member_type: VOTER last_known_addr { host: "127.14.147.195" port: 35113 } } }
I20260929 22:50:45.575798 22951 tablet_replica.cc:442] T 925fab2279cf420b983720a7da5b8421 P 60d7fd130c0f46ea83489a52a291a07a: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 22:50:45.575920 22958 mvcc.cc:205] Tried to move back new op lower bound from 7334798317867646976 to 7334798317839368192. Current Snapshot: MvccSnapshot[applied={T|T < 7334798317867646976 or (T in {7334798317867646976})}]
I20260929 22:50:45.576193 22962 tablet_replica.cc:439] T 925fab2279cf420b983720a7da5b8421 P 1144095f691a4cac85b7a4890ae1ae78: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "1144095f691a4cac85b7a4890ae1ae78" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a46aea30b8a8431189d809e841b82f6e" member_type: VOTER last_known_addr { host: "127.14.147.193" port: 33167 } } peers { permanent_uuid: "60d7fd130c0f46ea83489a52a291a07a" member_type: VOTER last_known_addr { host: "127.14.147.194" port: 44459 } } peers { permanent_uuid: "1144095f691a4cac85b7a4890ae1ae78" member_type: VOTER last_known_addr { host: "127.14.147.195" port: 35113 } } }
I20260929 22:50:45.576274 22962 tablet_replica.cc:442] T 925fab2279cf420b983720a7da5b8421 P 1144095f691a4cac85b7a4890ae1ae78: This TxnStatusTablet replica's current role is: LEADER
I20260929 22:50:45.576617 22959 mvcc.cc:205] Tried to move back new op lower bound from 7334798317867646976 to 7334798317839368192. Current Snapshot: MvccSnapshot[applied={T|T < 7334798317867646976}]
I20260929 22:50:45.576738 22950 tablet_replica.cc:439] T 925fab2279cf420b983720a7da5b8421 P a46aea30b8a8431189d809e841b82f6e: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "1144095f691a4cac85b7a4890ae1ae78" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a46aea30b8a8431189d809e841b82f6e" member_type: VOTER last_known_addr { host: "127.14.147.193" port: 33167 } } peers { permanent_uuid: "60d7fd130c0f46ea83489a52a291a07a" member_type: VOTER last_known_addr { host: "127.14.147.194" port: 44459 } } peers { permanent_uuid: "1144095f691a4cac85b7a4890ae1ae78" member_type: VOTER last_known_addr { host: "127.14.147.195" port: 35113 } } }
I20260929 22:50:45.576826 22950 tablet_replica.cc:442] T 925fab2279cf420b983720a7da5b8421 P a46aea30b8a8431189d809e841b82f6e: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 22:50:45.578797 22943 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:50:45.576315 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46808 (local address 127.14.147.194:44459)
0929 22:50:45.576358 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0929 22:50:45.576360 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 22:50:45.576369 (+ 9us) server_negotiation.cc:407] Connection header received
0929 22:50:45.576471 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 22:50:45.576473 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 22:50:45.576491 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 22:50:45.576549 (+ 58us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 22:50:45.576712 (+ 163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:50:45.577320 (+ 608us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:50:45.577992 (+ 672us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:50:45.578310 (+ 318us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:50:45.578383 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 22:50:45.578473 (+ 90us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 22:50:45.578609 (+ 136us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 22:50:45.578669 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0929 22:50:45.578688 (+ 19us) server_negotiation.cc:299] Negotiation successful
0929 22:50:45.578707 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260929 22:50:45.583796 22941 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:50:45.581284 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.147.194:54057 (local address 127.14.147.195:35113)
0929 22:50:45.581323 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0929 22:50:45.581325 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 22:50:45.581334 (+ 9us) server_negotiation.cc:407] Connection header received
0929 22:50:45.581390 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 22:50:45.581391 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 22:50:45.581401 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 22:50:45.581435 (+ 34us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 22:50:45.581553 (+ 118us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:50:45.582043 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:50:45.583111 (+ 1068us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:50:45.583303 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:50:45.583364 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 22:50:45.583584 (+ 220us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 22:50:45.583612 (+ 28us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 22:50:45.583649 (+ 37us) server_negotiation.cc:1035] Waiting for connection context
0929 22:50:45.583712 (+ 63us) server_negotiation.cc:299] Negotiation successful
0929 22:50:45.583731 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260929 22:50:45.583832 22781 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:50:45.581207 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.147.195:35113 (local address 127.14.147.194:54057)
0929 22:50:45.581270 (+ 63us) negotiation.cc:107] Waiting for socket to connect
0929 22:50:45.581281 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0929 22:50:45.581332 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 22:50:45.581441 (+ 109us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 22:50:45.581441 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 22:50:45.581442 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0929 22:50:45.581501 (+ 59us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 22:50:45.581502 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 22:50:45.582107 (+ 605us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 22:50:45.582107 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 22:50:45.583012 (+ 905us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 22:50:45.583014 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 22:50:45.583372 (+ 358us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 22:50:45.583372 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 22:50:45.583489 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 22:50:45.583509 (+ 20us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0929 22:50:45.583662 (+ 153us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0929 22:50:45.583663 (+ 1us) client_negotiation.cc:769] Sending connection context
0929 22:50:45.583709 (+ 46us) client_negotiation.cc:240] Negotiation successful
0929 22:50:45.583754 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":31}
I20260929 22:50:45.585964 22585 raft_consensus.cc:1261] T c72268163c6c48fe8a5ff982b28d7408 P a46aea30b8a8431189d809e841b82f6e [term 1 FOLLOWER]: Refusing update from remote peer 60d7fd130c0f46ea83489a52a291a07a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 22:50:45.586062 22870 raft_consensus.cc:1261] T c72268163c6c48fe8a5ff982b28d7408 P 1144095f691a4cac85b7a4890ae1ae78 [term 1 FOLLOWER]: Refusing update from remote peer 60d7fd130c0f46ea83489a52a291a07a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 22:50:45.586371 22974 consensus_queue.cc:1035] T c72268163c6c48fe8a5ff982b28d7408 P 60d7fd130c0f46ea83489a52a291a07a [LEADER]: Connected to new peer: Peer: permanent_uuid: "1144095f691a4cac85b7a4890ae1ae78" member_type: VOTER last_known_addr { host: "127.14.147.195" port: 35113 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 22:50:45.586491 22951 consensus_queue.cc:1035] T c72268163c6c48fe8a5ff982b28d7408 P 60d7fd130c0f46ea83489a52a291a07a [LEADER]: Connected to new peer: Peer: permanent_uuid: "a46aea30b8a8431189d809e841b82f6e" member_type: VOTER last_known_addr { host: "127.14.147.193" port: 33167 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 22:50:45.594913 22943 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:50:45.592321 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.147.195:45571 (local address 127.14.147.194:44459)
0929 22:50:45.592360 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0929 22:50:45.592362 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 22:50:45.592370 (+ 8us) server_negotiation.cc:407] Connection header received
0929 22:50:45.592423 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 22:50:45.592424 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 22:50:45.592433 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 22:50:45.592490 (+ 57us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 22:50:45.592581 (+ 91us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:50:45.593315 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:50:45.594178 (+ 863us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:50:45.594451 (+ 273us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:50:45.594506 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 22:50:45.594669 (+ 163us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 22:50:45.594705 (+ 36us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 22:50:45.594748 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0929 22:50:45.594812 (+ 64us) server_negotiation.cc:299] Negotiation successful
0929 22:50:45.594833 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260929 22:50:45.594914 22922 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:50:45.592262 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.147.194:44459 (local address 127.14.147.195:45571)
0929 22:50:45.592325 (+ 63us) negotiation.cc:107] Waiting for socket to connect
0929 22:50:45.592338 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 22:50:45.592375 (+ 37us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 22:50:45.592494 (+ 119us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 22:50:45.592494 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 22:50:45.592496 (+ 2us) client_negotiation.cc:189] Negotiated authn=TOKEN
0929 22:50:45.592537 (+ 41us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 22:50:45.592538 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 22:50:45.593392 (+ 854us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 22:50:45.593393 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 22:50:45.594059 (+ 666us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 22:50:45.594063 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 22:50:45.594517 (+ 454us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 22:50:45.594518 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 22:50:45.594580 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 22:50:45.594597 (+ 17us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0929 22:50:45.594755 (+ 158us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0929 22:50:45.594757 (+ 2us) client_negotiation.cc:769] Sending connection context
0929 22:50:45.594799 (+ 42us) client_negotiation.cc:240] Negotiation successful
0929 22:50:45.594849 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":21}
I20260929 22:50:45.601521 22430 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:46512:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 22:50:45.601639 22430 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:46512:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 22:50:45.602198 22430 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 7c99fb093ec947debdad2c01a7fd8027: Sending DeleteTablet for 3 replicas of tablet c72268163c6c48fe8a5ff982b28d7408
I20260929 22:50:45.602520 22709 tablet_service.cc:1514] Processing DeleteTablet for tablet c72268163c6c48fe8a5ff982b28d7408 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 22:50:45 UTC) from {username='oryx', principal='oryx/127.14.147.254@KRBTEST.COM'} at 127.0.0.1:46792
I20260929 22:50:45.602528 22568 tablet_service.cc:1514] Processing DeleteTablet for tablet c72268163c6c48fe8a5ff982b28d7408 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 22:50:45 UTC) from {username='oryx', principal='oryx/127.14.147.254@KRBTEST.COM'} at 127.0.0.1:41272
I20260929 22:50:45.602623 22968 tablet_replica.cc:330] T c72268163c6c48fe8a5ff982b28d7408 P 60d7fd130c0f46ea83489a52a291a07a: stopping tablet replica
I20260929 22:50:45.602624 22969 tablet_replica.cc:330] T c72268163c6c48fe8a5ff982b28d7408 P a46aea30b8a8431189d809e841b82f6e: stopping tablet replica
I20260929 22:50:45.602679 22968 raft_consensus.cc:2229] T c72268163c6c48fe8a5ff982b28d7408 P 60d7fd130c0f46ea83489a52a291a07a [term 1 LEADER]: Raft consensus shutting down.
I20260929 22:50:45.602679 22969 raft_consensus.cc:2229] T c72268163c6c48fe8a5ff982b28d7408 P a46aea30b8a8431189d809e841b82f6e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 22:50:45.602717 22969 raft_consensus.cc:2258] T c72268163c6c48fe8a5ff982b28d7408 P a46aea30b8a8431189d809e841b82f6e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 22:50:45.602772 22968 raft_consensus.cc:2258] T c72268163c6c48fe8a5ff982b28d7408 P 60d7fd130c0f46ea83489a52a291a07a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 22:50:45.603062 22968 ts_tablet_manager.cc:1905] T c72268163c6c48fe8a5ff982b28d7408 P 60d7fd130c0f46ea83489a52a291a07a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 22:50:45.603607 22835 tablet_service.cc:1514] Processing DeleteTablet for tablet c72268163c6c48fe8a5ff982b28d7408 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 22:50:45 UTC) from {username='oryx', principal='oryx/127.14.147.254@KRBTEST.COM'} at 127.0.0.1:49068
I20260929 22:50:45.603695 22967 tablet_replica.cc:330] T c72268163c6c48fe8a5ff982b28d7408 P 1144095f691a4cac85b7a4890ae1ae78: stopping tablet replica
I20260929 22:50:45.603754 22967 raft_consensus.cc:2229] T c72268163c6c48fe8a5ff982b28d7408 P 1144095f691a4cac85b7a4890ae1ae78 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 22:50:45.603798 22967 raft_consensus.cc:2258] T c72268163c6c48fe8a5ff982b28d7408 P 1144095f691a4cac85b7a4890ae1ae78 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 22:50:45.604125 22967 ts_tablet_manager.cc:1905] T c72268163c6c48fe8a5ff982b28d7408 P 1144095f691a4cac85b7a4890ae1ae78: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 22:50:45.604679 22968 ts_tablet_manager.cc:1918] T c72268163c6c48fe8a5ff982b28d7408 P 60d7fd130c0f46ea83489a52a291a07a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260929 22:50:45.604741 22968 log.cc:1196] T c72268163c6c48fe8a5ff982b28d7408 P 60d7fd130c0f46ea83489a52a291a07a: Deleting WAL directory at /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790722213368738-14927-0/minicluster-data/ts-1/wal/wals/c72268163c6c48fe8a5ff982b28d7408
I20260929 22:50:45.604941 22968 ts_tablet_manager.cc:1939] T c72268163c6c48fe8a5ff982b28d7408 P 60d7fd130c0f46ea83489a52a291a07a: Deleting consensus metadata
I20260929 22:50:45.605299 22417 catalog_manager.cc:4841] TS 60d7fd130c0f46ea83489a52a291a07a (127.14.147.194:44459): tablet c72268163c6c48fe8a5ff982b28d7408 (table test-table [id=35d3e22dab834129a5bea999b2a686a9]) successfully deleted
I20260929 22:50:45.605600 22969 ts_tablet_manager.cc:1905] T c72268163c6c48fe8a5ff982b28d7408 P a46aea30b8a8431189d809e841b82f6e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 22:50:45.606719 22969 ts_tablet_manager.cc:1918] T c72268163c6c48fe8a5ff982b28d7408 P a46aea30b8a8431189d809e841b82f6e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260929 22:50:45.606775 22969 log.cc:1196] T c72268163c6c48fe8a5ff982b28d7408 P a46aea30b8a8431189d809e841b82f6e: Deleting WAL directory at /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790722213368738-14927-0/minicluster-data/ts-0/wal/wals/c72268163c6c48fe8a5ff982b28d7408
I20260929 22:50:45.606963 22969 ts_tablet_manager.cc:1939] T c72268163c6c48fe8a5ff982b28d7408 P a46aea30b8a8431189d809e841b82f6e: Deleting consensus metadata
I20260929 22:50:45.607296 22417 catalog_manager.cc:4841] TS a46aea30b8a8431189d809e841b82f6e (127.14.147.193:33167): tablet c72268163c6c48fe8a5ff982b28d7408 (table test-table [id=35d3e22dab834129a5bea999b2a686a9]) successfully deleted
I20260929 22:50:45.607648 22967 ts_tablet_manager.cc:1918] T c72268163c6c48fe8a5ff982b28d7408 P 1144095f691a4cac85b7a4890ae1ae78: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260929 22:50:45.607712 22967 log.cc:1196] T c72268163c6c48fe8a5ff982b28d7408 P 1144095f691a4cac85b7a4890ae1ae78: Deleting WAL directory at /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790722213368738-14927-0/minicluster-data/ts-2/wal/wals/c72268163c6c48fe8a5ff982b28d7408
I20260929 22:50:45.607914 22967 ts_tablet_manager.cc:1939] T c72268163c6c48fe8a5ff982b28d7408 P 1144095f691a4cac85b7a4890ae1ae78: Deleting consensus metadata
I20260929 22:50:45.608225 22416 catalog_manager.cc:4841] TS 1144095f691a4cac85b7a4890ae1ae78 (127.14.147.195:35113): tablet c72268163c6c48fe8a5ff982b28d7408 (table test-table [id=35d3e22dab834129a5bea999b2a686a9]) successfully deleted
Sep 29 22:50:45 dist-test-slave-98mk krb5kdc[22385](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790722245, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 29 22:50:45 dist-test-slave-98mk krb5kdc[22385](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.14.147.254@KRBTEST.COM, Server not found in Kerberos database
Sep 29 22:50:45 dist-test-slave-98mk krb5kdc[22385](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.14.147.254@KRBTEST.COM, Server not found in Kerberos database
W20260929 22:50:45.634887 22480 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 22:50:45.627908 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46532 (local address 127.14.147.254:33005)
0929 22:50:45.627959 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0929 22:50:45.627961 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 22:50:45.628137 (+ 176us) server_negotiation.cc:407] Connection header received
0929 22:50:45.628197 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 22:50:45.628198 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 22:50:45.628210 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 22:50:45.628274 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0929 22:50:45.628993 (+ 719us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:50:45.629779 (+ 786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:50:45.630948 (+ 1169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:50:45.631242 (+ 294us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:50:45.631322 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 22:50:45.634808 (+ 3486us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:46532: BlockingRecv error: recv got EOF from 127.0.0.1:46532 (error 108)
Metrics: {"server-negotiator.queue_time_us":23}
Sep 29 22:50:45 dist-test-slave-98mk krb5kdc[22385](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790722245, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.14.147.254@KRBTEST.COM
I20260929 22:50:45.644690 22480 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:50:45.636045 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46540 (local address 127.14.147.254:33005)
0929 22:50:45.636085 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0929 22:50:45.636087 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 22:50:45.636311 (+ 224us) server_negotiation.cc:407] Connection header received
0929 22:50:45.636364 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 22:50:45.636365 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 22:50:45.636375 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 22:50:45.636433 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0929 22:50:45.637003 (+ 570us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:50:45.637779 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:50:45.638666 (+ 887us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:50:45.638969 (+ 303us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:50:45.639034 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 22:50:45.641835 (+ 2801us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 22:50:45.641837 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 22:50:45.641838 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 22:50:45.641847 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0929 22:50:45.643465 (+ 1618us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 22:50:45.643871 (+ 406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 22:50:45.643872 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 22:50:45.643873 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 22:50:45.643953 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 22:50:45.644130 (+ 177us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 22:50:45.644131 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 22:50:45.644131 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 22:50:45.644266 (+ 135us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 22:50:45.644372 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0929 22:50:45.644498 (+ 126us) server_negotiation.cc:299] Negotiation successful
0929 22:50:45.644584 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260929 22:50:45.646795 22430 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46540:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 22:50:45.647001 22430 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 22:50:45.648582 22568 tablet_service.cc:1467] Processing CreateTablet for tablet 31a4c5ce55d244828068d0aedc66ebd8 (DEFAULT_TABLE table=test-table [id=04fec2465a004f828d2ed53eb0d4b6a7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 22:50:45.648646 22709 tablet_service.cc:1467] Processing CreateTablet for tablet 31a4c5ce55d244828068d0aedc66ebd8 (DEFAULT_TABLE table=test-table [id=04fec2465a004f828d2ed53eb0d4b6a7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 22:50:45.648736 22568 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 31a4c5ce55d244828068d0aedc66ebd8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 22:50:45.648777 22709 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 31a4c5ce55d244828068d0aedc66ebd8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 22:50:45.650164 22946 tablet_bootstrap.cc:492] T 31a4c5ce55d244828068d0aedc66ebd8 P 60d7fd130c0f46ea83489a52a291a07a: Bootstrap starting.
I20260929 22:50:45.650287 22835 tablet_service.cc:1467] Processing CreateTablet for tablet 31a4c5ce55d244828068d0aedc66ebd8 (DEFAULT_TABLE table=test-table [id=04fec2465a004f828d2ed53eb0d4b6a7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 22:50:45.650408 22835 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 31a4c5ce55d244828068d0aedc66ebd8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 22:50:45.650607 22946 tablet_bootstrap.cc:654] T 31a4c5ce55d244828068d0aedc66ebd8 P 60d7fd130c0f46ea83489a52a291a07a: Neither blocks nor log segments found. Creating new log.
I20260929 22:50:45.651499 22945 tablet_bootstrap.cc:492] T 31a4c5ce55d244828068d0aedc66ebd8 P a46aea30b8a8431189d809e841b82f6e: Bootstrap starting.
I20260929 22:50:45.651614 22946 tablet_bootstrap.cc:492] T 31a4c5ce55d244828068d0aedc66ebd8 P 60d7fd130c0f46ea83489a52a291a07a: No bootstrap required, opened a new log
I20260929 22:50:45.651657 22946 ts_tablet_manager.cc:1397] T 31a4c5ce55d244828068d0aedc66ebd8 P 60d7fd130c0f46ea83489a52a291a07a: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260929 22:50:45.651800 22946 raft_consensus.cc:348] T 31a4c5ce55d244828068d0aedc66ebd8 P 60d7fd130c0f46ea83489a52a291a07a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a46aea30b8a8431189d809e841b82f6e" member_type: VOTER last_known_addr { host: "127.14.147.193" port: 33167 } } peers { permanent_uuid: "60d7fd130c0f46ea83489a52a291a07a" member_type: VOTER last_known_addr { host: "127.14.147.194" port: 44459 } } peers { permanent_uuid: "1144095f691a4cac85b7a4890ae1ae78" member_type: VOTER last_known_addr { host: "127.14.147.195" port: 35113 } }
I20260929 22:50:45.651865 22946 raft_consensus.cc:374] T 31a4c5ce55d244828068d0aedc66ebd8 P 60d7fd130c0f46ea83489a52a291a07a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 22:50:45.651883 22946 raft_consensus.cc:729] T 31a4c5ce55d244828068d0aedc66ebd8 P 60d7fd130c0f46ea83489a52a291a07a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 60d7fd130c0f46ea83489a52a291a07a, State: Initialized, Role: FOLLOWER
I20260929 22:50:45.651930 22945 tablet_bootstrap.cc:654] T 31a4c5ce55d244828068d0aedc66ebd8 P a46aea30b8a8431189d809e841b82f6e: Neither blocks nor log segments found. Creating new log.
I20260929 22:50:45.651937 22946 consensus_queue.cc:260] T 31a4c5ce55d244828068d0aedc66ebd8 P 60d7fd130c0f46ea83489a52a291a07a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a46aea30b8a8431189d809e841b82f6e" member_type: VOTER last_known_addr { host: "127.14.147.193" port: 33167 } } peers { permanent_uuid: "60d7fd130c0f46ea83489a52a291a07a" member_type: VOTER last_known_addr { host: "127.14.147.194" port: 44459 } } peers { permanent_uuid: "1144095f691a4cac85b7a4890ae1ae78" member_type: VOTER last_known_addr { host: "127.14.147.195" port: 35113 } }
I20260929 22:50:45.652025 22946 ts_tablet_manager.cc:1428] T 31a4c5ce55d244828068d0aedc66ebd8 P 60d7fd130c0f46ea83489a52a291a07a: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260929 22:50:45.652981 22944 tablet_bootstrap.cc:492] T 31a4c5ce55d244828068d0aedc66ebd8 P 1144095f691a4cac85b7a4890ae1ae78: Bootstrap starting.
I20260929 22:50:45.653311 22944 tablet_bootstrap.cc:654] T 31a4c5ce55d244828068d0aedc66ebd8 P 1144095f691a4cac85b7a4890ae1ae78: Neither blocks nor log segments found. Creating new log.
I20260929 22:50:45.653514 22945 tablet_bootstrap.cc:492] T 31a4c5ce55d244828068d0aedc66ebd8 P a46aea30b8a8431189d809e841b82f6e: No bootstrap required, opened a new log
I20260929 22:50:45.653559 22945 ts_tablet_manager.cc:1397] T 31a4c5ce55d244828068d0aedc66ebd8 P a46aea30b8a8431189d809e841b82f6e: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260929 22:50:45.653707 22945 raft_consensus.cc:348] T 31a4c5ce55d244828068d0aedc66ebd8 P a46aea30b8a8431189d809e841b82f6e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a46aea30b8a8431189d809e841b82f6e" member_type: VOTER last_known_addr { host: "127.14.147.193" port: 33167 } } peers { permanent_uuid: "60d7fd130c0f46ea83489a52a291a07a" member_type: VOTER last_known_addr { host: "127.14.147.194" port: 44459 } } peers { permanent_uuid: "1144095f691a4cac85b7a4890ae1ae78" member_type: VOTER last_known_addr { host: "127.14.147.195" port: 35113 } }
I20260929 22:50:45.653790 22945 raft_consensus.cc:374] T 31a4c5ce55d244828068d0aedc66ebd8 P a46aea30b8a8431189d809e841b82f6e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 22:50:45.653831 22945 raft_consensus.cc:729] T 31a4c5ce55d244828068d0aedc66ebd8 P a46aea30b8a8431189d809e841b82f6e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a46aea30b8a8431189d809e841b82f6e, State: Initialized, Role: FOLLOWER
I20260929 22:50:45.653872 22945 consensus_queue.cc:260] T 31a4c5ce55d244828068d0aedc66ebd8 P a46aea30b8a8431189d809e841b82f6e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a46aea30b8a8431189d809e841b82f6e" member_type: VOTER last_known_addr { host: "127.14.147.193" port: 33167 } } peers { permanent_uuid: "60d7fd130c0f46ea83489a52a291a07a" member_type: VOTER last_known_addr { host: "127.14.147.194" port: 44459 } } peers { permanent_uuid: "1144095f691a4cac85b7a4890ae1ae78" member_type: VOTER last_known_addr { host: "127.14.147.195" port: 35113 } }
I20260929 22:50:45.653970 22945 ts_tablet_manager.cc:1428] T 31a4c5ce55d244828068d0aedc66ebd8 P a46aea30b8a8431189d809e841b82f6e: Time spent starting tablet: real 0.000s user 0.000s sys 0.001s
I20260929 22:50:45.654045 22944 tablet_bootstrap.cc:492] T 31a4c5ce55d244828068d0aedc66ebd8 P 1144095f691a4cac85b7a4890ae1ae78: No bootstrap required, opened a new log
I20260929 22:50:45.654093 22944 ts_tablet_manager.cc:1397] T 31a4c5ce55d244828068d0aedc66ebd8 P 1144095f691a4cac85b7a4890ae1ae78: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260929 22:50:45.654227 22944 raft_consensus.cc:348] T 31a4c5ce55d244828068d0aedc66ebd8 P 1144095f691a4cac85b7a4890ae1ae78 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a46aea30b8a8431189d809e841b82f6e" member_type: VOTER last_known_addr { host: "127.14.147.193" port: 33167 } } peers { permanent_uuid: "60d7fd130c0f46ea83489a52a291a07a" member_type: VOTER last_known_addr { host: "127.14.147.194" port: 44459 } } peers { permanent_uuid: "1144095f691a4cac85b7a4890ae1ae78" member_type: VOTER last_known_addr { host: "127.14.147.195" port: 35113 } }
I20260929 22:50:45.654284 22944 raft_consensus.cc:374] T 31a4c5ce55d244828068d0aedc66ebd8 P 1144095f691a4cac85b7a4890ae1ae78 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 22:50:45.654300 22944 raft_consensus.cc:729] T 31a4c5ce55d244828068d0aedc66ebd8 P 1144095f691a4cac85b7a4890ae1ae78 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1144095f691a4cac85b7a4890ae1ae78, State: Initialized, Role: FOLLOWER
I20260929 22:50:45.654341 22944 consensus_queue.cc:260] T 31a4c5ce55d244828068d0aedc66ebd8 P 1144095f691a4cac85b7a4890ae1ae78 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a46aea30b8a8431189d809e841b82f6e" member_type: VOTER last_known_addr { host: "127.14.147.193" port: 33167 } } peers { permanent_uuid: "60d7fd130c0f46ea83489a52a291a07a" member_type: VOTER last_known_addr { host: "127.14.147.194" port: 44459 } } peers { permanent_uuid: "1144095f691a4cac85b7a4890ae1ae78" member_type: VOTER last_known_addr { host: "127.14.147.195" port: 35113 } }
I20260929 22:50:45.654415 22944 ts_tablet_manager.cc:1428] T 31a4c5ce55d244828068d0aedc66ebd8 P 1144095f691a4cac85b7a4890ae1ae78: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260929 22:50:45.733959 22954 raft_consensus.cc:482] T 31a4c5ce55d244828068d0aedc66ebd8 P 1144095f691a4cac85b7a4890ae1ae78 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 22:50:45.734062 22954 raft_consensus.cc:504] T 31a4c5ce55d244828068d0aedc66ebd8 P 1144095f691a4cac85b7a4890ae1ae78 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a46aea30b8a8431189d809e841b82f6e" member_type: VOTER last_known_addr { host: "127.14.147.193" port: 33167 } } peers { permanent_uuid: "60d7fd130c0f46ea83489a52a291a07a" member_type: VOTER last_known_addr { host: "127.14.147.194" port: 44459 } } peers { permanent_uuid: "1144095f691a4cac85b7a4890ae1ae78" member_type: VOTER last_known_addr { host: "127.14.147.195" port: 35113 } }
I20260929 22:50:45.734241 22954 leader_election.cc:288] T 31a4c5ce55d244828068d0aedc66ebd8 P 1144095f691a4cac85b7a4890ae1ae78 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a46aea30b8a8431189d809e841b82f6e (127.14.147.193:33167), 60d7fd130c0f46ea83489a52a291a07a (127.14.147.194:44459)
I20260929 22:50:45.734494 22729 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "31a4c5ce55d244828068d0aedc66ebd8" candidate_uuid: "1144095f691a4cac85b7a4890ae1ae78" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "60d7fd130c0f46ea83489a52a291a07a" is_pre_election: true
I20260929 22:50:45.734607 22729 raft_consensus.cc:2454] T 31a4c5ce55d244828068d0aedc66ebd8 P 60d7fd130c0f46ea83489a52a291a07a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1144095f691a4cac85b7a4890ae1ae78 in term 0.
I20260929 22:50:45.734587 22585 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "31a4c5ce55d244828068d0aedc66ebd8" candidate_uuid: "1144095f691a4cac85b7a4890ae1ae78" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a46aea30b8a8431189d809e841b82f6e" is_pre_election: true
I20260929 22:50:45.734673 22585 raft_consensus.cc:2454] T 31a4c5ce55d244828068d0aedc66ebd8 P a46aea30b8a8431189d809e841b82f6e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1144095f691a4cac85b7a4890ae1ae78 in term 0.
I20260929 22:50:45.734778 22803 leader_election.cc:302] T 31a4c5ce55d244828068d0aedc66ebd8 P 1144095f691a4cac85b7a4890ae1ae78 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1144095f691a4cac85b7a4890ae1ae78, 60d7fd130c0f46ea83489a52a291a07a; no voters:
I20260929 22:50:45.734899 22954 raft_consensus.cc:2781] T 31a4c5ce55d244828068d0aedc66ebd8 P 1144095f691a4cac85b7a4890ae1ae78 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 22:50:45.734938 22954 raft_consensus.cc:482] T 31a4c5ce55d244828068d0aedc66ebd8 P 1144095f691a4cac85b7a4890ae1ae78 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 22:50:45.734952 22954 raft_consensus.cc:3037] T 31a4c5ce55d244828068d0aedc66ebd8 P 1144095f691a4cac85b7a4890ae1ae78 [term 0 FOLLOWER]: Advancing to term 1
I20260929 22:50:45.735651 22954 raft_consensus.cc:504] T 31a4c5ce55d244828068d0aedc66ebd8 P 1144095f691a4cac85b7a4890ae1ae78 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a46aea30b8a8431189d809e841b82f6e" member_type: VOTER last_known_addr { host: "127.14.147.193" port: 33167 } } peers { permanent_uuid: "60d7fd130c0f46ea83489a52a291a07a" member_type: VOTER last_known_addr { host: "127.14.147.194" port: 44459 } } peers { permanent_uuid: "1144095f691a4cac85b7a4890ae1ae78" member_type: VOTER last_known_addr { host: "127.14.147.195" port: 35113 } }
I20260929 22:50:45.735764 22954 leader_election.cc:288] T 31a4c5ce55d244828068d0aedc66ebd8 P 1144095f691a4cac85b7a4890ae1ae78 [CANDIDATE]: Term 1 election: Requested vote from peers a46aea30b8a8431189d809e841b82f6e (127.14.147.193:33167), 60d7fd130c0f46ea83489a52a291a07a (127.14.147.194:44459)
I20260929 22:50:45.735948 22585 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "31a4c5ce55d244828068d0aedc66ebd8" candidate_uuid: "1144095f691a4cac85b7a4890ae1ae78" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a46aea30b8a8431189d809e841b82f6e"
I20260929 22:50:45.736019 22585 raft_consensus.cc:3037] T 31a4c5ce55d244828068d0aedc66ebd8 P a46aea30b8a8431189d809e841b82f6e [term 0 FOLLOWER]: Advancing to term 1
I20260929 22:50:45.735968 22729 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "31a4c5ce55d244828068d0aedc66ebd8" candidate_uuid: "1144095f691a4cac85b7a4890ae1ae78" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "60d7fd130c0f46ea83489a52a291a07a"
I20260929 22:50:45.736049 22729 raft_consensus.cc:3037] T 31a4c5ce55d244828068d0aedc66ebd8 P 60d7fd130c0f46ea83489a52a291a07a [term 0 FOLLOWER]: Advancing to term 1
I20260929 22:50:45.736637 22585 raft_consensus.cc:2454] T 31a4c5ce55d244828068d0aedc66ebd8 P a46aea30b8a8431189d809e841b82f6e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1144095f691a4cac85b7a4890ae1ae78 in term 1.
I20260929 22:50:45.736685 22729 raft_consensus.cc:2454] T 31a4c5ce55d244828068d0aedc66ebd8 P 60d7fd130c0f46ea83489a52a291a07a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1144095f691a4cac85b7a4890ae1ae78 in term 1.
I20260929 22:50:45.736853 22803 leader_election.cc:302] T 31a4c5ce55d244828068d0aedc66ebd8 P 1144095f691a4cac85b7a4890ae1ae78 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1144095f691a4cac85b7a4890ae1ae78, a46aea30b8a8431189d809e841b82f6e; no voters:
I20260929 22:50:45.736951 22954 raft_consensus.cc:2781] T 31a4c5ce55d244828068d0aedc66ebd8 P 1144095f691a4cac85b7a4890ae1ae78 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 22:50:45.737004 22954 raft_consensus.cc:686] T 31a4c5ce55d244828068d0aedc66ebd8 P 1144095f691a4cac85b7a4890ae1ae78 [term 1 LEADER]: Becoming Leader. State: Replica: 1144095f691a4cac85b7a4890ae1ae78, State: Running, Role: LEADER
I20260929 22:50:45.737054 22954 consensus_queue.cc:237] T 31a4c5ce55d244828068d0aedc66ebd8 P 1144095f691a4cac85b7a4890ae1ae78 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a46aea30b8a8431189d809e841b82f6e" member_type: VOTER last_known_addr { host: "127.14.147.193" port: 33167 } } peers { permanent_uuid: "60d7fd130c0f46ea83489a52a291a07a" member_type: VOTER last_known_addr { host: "127.14.147.194" port: 44459 } } peers { permanent_uuid: "1144095f691a4cac85b7a4890ae1ae78" member_type: VOTER last_known_addr { host: "127.14.147.195" port: 35113 } }
I20260929 22:50:45.737641 22428 catalog_manager.cc:5495] T 31a4c5ce55d244828068d0aedc66ebd8 P 1144095f691a4cac85b7a4890ae1ae78 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1144095f691a4cac85b7a4890ae1ae78 (127.14.147.195). New cstate: current_term: 1 leader_uuid: "1144095f691a4cac85b7a4890ae1ae78" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a46aea30b8a8431189d809e841b82f6e" member_type: VOTER last_known_addr { host: "127.14.147.193" port: 33167 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "60d7fd130c0f46ea83489a52a291a07a" member_type: VOTER last_known_addr { host: "127.14.147.194" port: 44459 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1144095f691a4cac85b7a4890ae1ae78" member_type: VOTER last_known_addr { host: "127.14.147.195" port: 35113 } health_report { overall_health: HEALTHY } } }
I20260929 22:50:45.750229 22941 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:50:45.748108 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49110 (local address 127.14.147.195:35113)
0929 22:50:45.748154 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0929 22:50:45.748156 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 22:50:45.748166 (+ 10us) server_negotiation.cc:407] Connection header received
0929 22:50:45.748194 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 22:50:45.748194 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 22:50:45.748203 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 22:50:45.748277 (+ 74us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 22:50:45.748374 (+ 97us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:50:45.748827 (+ 453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:50:45.749564 (+ 737us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:50:45.749750 (+ 186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:50:45.749797 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 22:50:45.749990 (+ 193us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 22:50:45.750030 (+ 40us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 22:50:45.750089 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0929 22:50:45.750129 (+ 40us) server_negotiation.cc:299] Negotiation successful
0929 22:50:45.750151 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260929 22:50:45.750927 22729 raft_consensus.cc:1261] T 31a4c5ce55d244828068d0aedc66ebd8 P 60d7fd130c0f46ea83489a52a291a07a [term 1 FOLLOWER]: Refusing update from remote peer 1144095f691a4cac85b7a4890ae1ae78: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 22:50:45.750938 22585 raft_consensus.cc:1261] T 31a4c5ce55d244828068d0aedc66ebd8 P a46aea30b8a8431189d809e841b82f6e [term 1 FOLLOWER]: Refusing update from remote peer 1144095f691a4cac85b7a4890ae1ae78: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 22:50:45.751151 22954 consensus_queue.cc:1035] T 31a4c5ce55d244828068d0aedc66ebd8 P 1144095f691a4cac85b7a4890ae1ae78 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a46aea30b8a8431189d809e841b82f6e" member_type: VOTER last_known_addr { host: "127.14.147.193" port: 33167 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 22:50:45.751247 22952 consensus_queue.cc:1035] T 31a4c5ce55d244828068d0aedc66ebd8 P 1144095f691a4cac85b7a4890ae1ae78 [LEADER]: Connected to new peer: Peer: permanent_uuid: "60d7fd130c0f46ea83489a52a291a07a" member_type: VOTER last_known_addr { host: "127.14.147.194" port: 44459 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 22:50:45.753929 22428 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46540:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 22:50:45.754016 22428 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46540:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 22:50:45.754598 22428 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 7c99fb093ec947debdad2c01a7fd8027: Sending DeleteTablet for 3 replicas of tablet 31a4c5ce55d244828068d0aedc66ebd8
I20260929 22:50:45.754869 22835 tablet_service.cc:1514] Processing DeleteTablet for tablet 31a4c5ce55d244828068d0aedc66ebd8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 22:50:45 UTC) from {username='oryx', principal='oryx/127.14.147.254@KRBTEST.COM'} at 127.0.0.1:49068
I20260929 22:50:45.754868 22568 tablet_service.cc:1514] Processing DeleteTablet for tablet 31a4c5ce55d244828068d0aedc66ebd8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 22:50:45 UTC) from {username='oryx', principal='oryx/127.14.147.254@KRBTEST.COM'} at 127.0.0.1:41272
I20260929 22:50:45.754951 22969 tablet_replica.cc:330] T 31a4c5ce55d244828068d0aedc66ebd8 P a46aea30b8a8431189d809e841b82f6e: stopping tablet replica
I20260929 22:50:45.754953 22967 tablet_replica.cc:330] T 31a4c5ce55d244828068d0aedc66ebd8 P 1144095f691a4cac85b7a4890ae1ae78: stopping tablet replica
I20260929 22:50:45.755002 22967 raft_consensus.cc:2229] T 31a4c5ce55d244828068d0aedc66ebd8 P 1144095f691a4cac85b7a4890ae1ae78 [term 1 LEADER]: Raft consensus shutting down.
I20260929 22:50:45.755002 22969 raft_consensus.cc:2229] T 31a4c5ce55d244828068d0aedc66ebd8 P a46aea30b8a8431189d809e841b82f6e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 22:50:45.755043 22969 raft_consensus.cc:2258] T 31a4c5ce55d244828068d0aedc66ebd8 P a46aea30b8a8431189d809e841b82f6e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 22:50:45.755055 22967 raft_consensus.cc:2258] T 31a4c5ce55d244828068d0aedc66ebd8 P 1144095f691a4cac85b7a4890ae1ae78 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 22:50:45.755286 22967 ts_tablet_manager.cc:1905] T 31a4c5ce55d244828068d0aedc66ebd8 P 1144095f691a4cac85b7a4890ae1ae78: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 22:50:45.755280 22709 tablet_service.cc:1514] Processing DeleteTablet for tablet 31a4c5ce55d244828068d0aedc66ebd8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 22:50:45 UTC) from {username='oryx', principal='oryx/127.14.147.254@KRBTEST.COM'} at 127.0.0.1:46792
I20260929 22:50:45.755365 22968 tablet_replica.cc:330] T 31a4c5ce55d244828068d0aedc66ebd8 P 60d7fd130c0f46ea83489a52a291a07a: stopping tablet replica
I20260929 22:50:45.755445 22968 raft_consensus.cc:2229] T 31a4c5ce55d244828068d0aedc66ebd8 P 60d7fd130c0f46ea83489a52a291a07a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 22:50:45.755492 22968 raft_consensus.cc:2258] T 31a4c5ce55d244828068d0aedc66ebd8 P 60d7fd130c0f46ea83489a52a291a07a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 22:50:45.755548 22969 ts_tablet_manager.cc:1905] T 31a4c5ce55d244828068d0aedc66ebd8 P a46aea30b8a8431189d809e841b82f6e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 22:50:45.756628 22969 ts_tablet_manager.cc:1918] T 31a4c5ce55d244828068d0aedc66ebd8 P a46aea30b8a8431189d809e841b82f6e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 22:50:45.756647 22967 ts_tablet_manager.cc:1918] T 31a4c5ce55d244828068d0aedc66ebd8 P 1144095f691a4cac85b7a4890ae1ae78: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 22:50:45.756662 22968 ts_tablet_manager.cc:1905] T 31a4c5ce55d244828068d0aedc66ebd8 P 60d7fd130c0f46ea83489a52a291a07a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 22:50:45.756692 22967 log.cc:1196] T 31a4c5ce55d244828068d0aedc66ebd8 P 1144095f691a4cac85b7a4890ae1ae78: Deleting WAL directory at /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790722213368738-14927-0/minicluster-data/ts-2/wal/wals/31a4c5ce55d244828068d0aedc66ebd8
I20260929 22:50:45.756703 22969 log.cc:1196] T 31a4c5ce55d244828068d0aedc66ebd8 P a46aea30b8a8431189d809e841b82f6e: Deleting WAL directory at /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790722213368738-14927-0/minicluster-data/ts-0/wal/wals/31a4c5ce55d244828068d0aedc66ebd8
I20260929 22:50:45.756897 22969 ts_tablet_manager.cc:1939] T 31a4c5ce55d244828068d0aedc66ebd8 P a46aea30b8a8431189d809e841b82f6e: Deleting consensus metadata
I20260929 22:50:45.756927 22967 ts_tablet_manager.cc:1939] T 31a4c5ce55d244828068d0aedc66ebd8 P 1144095f691a4cac85b7a4890ae1ae78: Deleting consensus metadata
I20260929 22:50:45.757226 22416 catalog_manager.cc:4841] TS 1144095f691a4cac85b7a4890ae1ae78 (127.14.147.195:35113): tablet 31a4c5ce55d244828068d0aedc66ebd8 (table test-table [id=04fec2465a004f828d2ed53eb0d4b6a7]) successfully deleted
I20260929 22:50:45.757191 22428 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46540:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 22:50:45.757331 22428 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 22:50:45.758023 22968 ts_tablet_manager.cc:1918] T 31a4c5ce55d244828068d0aedc66ebd8 P 60d7fd130c0f46ea83489a52a291a07a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 22:50:45.758082 22968 log.cc:1196] T 31a4c5ce55d244828068d0aedc66ebd8 P 60d7fd130c0f46ea83489a52a291a07a: Deleting WAL directory at /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790722213368738-14927-0/minicluster-data/ts-1/wal/wals/31a4c5ce55d244828068d0aedc66ebd8
I20260929 22:50:45.758292 22968 ts_tablet_manager.cc:1939] T 31a4c5ce55d244828068d0aedc66ebd8 P 60d7fd130c0f46ea83489a52a291a07a: Deleting consensus metadata
I20260929 22:50:45.758625 22417 catalog_manager.cc:4841] TS a46aea30b8a8431189d809e841b82f6e (127.14.147.193:33167): tablet 31a4c5ce55d244828068d0aedc66ebd8 (table test-table [id=04fec2465a004f828d2ed53eb0d4b6a7]) successfully deleted
I20260929 22:50:45.758638 22568 tablet_service.cc:1467] Processing CreateTablet for tablet 75afa0d8359a4343a08ea58b91f5709a (DEFAULT_TABLE table=test-table [id=d3d8e98e5f3648fc992b480b878f7b48]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 22:50:45.758648 22709 tablet_service.cc:1467] Processing CreateTablet for tablet 75afa0d8359a4343a08ea58b91f5709a (DEFAULT_TABLE table=test-table [id=d3d8e98e5f3648fc992b480b878f7b48]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 22:50:45.758648 22835 tablet_service.cc:1467] Processing CreateTablet for tablet 75afa0d8359a4343a08ea58b91f5709a (DEFAULT_TABLE table=test-table [id=d3d8e98e5f3648fc992b480b878f7b48]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 22:50:45.758711 22417 catalog_manager.cc:4841] TS 60d7fd130c0f46ea83489a52a291a07a (127.14.147.194:44459): tablet 31a4c5ce55d244828068d0aedc66ebd8 (table test-table [id=04fec2465a004f828d2ed53eb0d4b6a7]) successfully deleted
I20260929 22:50:45.758745 22709 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 75afa0d8359a4343a08ea58b91f5709a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 22:50:45.758747 22835 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 75afa0d8359a4343a08ea58b91f5709a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 22:50:45.758754 22568 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 75afa0d8359a4343a08ea58b91f5709a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 22:50:45.759647 22944 tablet_bootstrap.cc:492] T 75afa0d8359a4343a08ea58b91f5709a P 1144095f691a4cac85b7a4890ae1ae78: Bootstrap starting.
I20260929 22:50:45.759941 22945 tablet_bootstrap.cc:492] T 75afa0d8359a4343a08ea58b91f5709a P a46aea30b8a8431189d809e841b82f6e: Bootstrap starting.
I20260929 22:50:45.759943 22946 tablet_bootstrap.cc:492] T 75afa0d8359a4343a08ea58b91f5709a P 60d7fd130c0f46ea83489a52a291a07a: Bootstrap starting.
I20260929 22:50:45.759961 22944 tablet_bootstrap.cc:654] T 75afa0d8359a4343a08ea58b91f5709a P 1144095f691a4cac85b7a4890ae1ae78: Neither blocks nor log segments found. Creating new log.
I20260929 22:50:45.760257 22946 tablet_bootstrap.cc:654] T 75afa0d8359a4343a08ea58b91f5709a P 60d7fd130c0f46ea83489a52a291a07a: Neither blocks nor log segments found. Creating new log.
I20260929 22:50:45.760499 22945 tablet_bootstrap.cc:654] T 75afa0d8359a4343a08ea58b91f5709a P a46aea30b8a8431189d809e841b82f6e: Neither blocks nor log segments found. Creating new log.
I20260929 22:50:45.760524 22944 tablet_bootstrap.cc:492] T 75afa0d8359a4343a08ea58b91f5709a P 1144095f691a4cac85b7a4890ae1ae78: No bootstrap required, opened a new log
I20260929 22:50:45.760557 22944 ts_tablet_manager.cc:1397] T 75afa0d8359a4343a08ea58b91f5709a P 1144095f691a4cac85b7a4890ae1ae78: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260929 22:50:45.760715 22944 raft_consensus.cc:348] T 75afa0d8359a4343a08ea58b91f5709a P 1144095f691a4cac85b7a4890ae1ae78 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a46aea30b8a8431189d809e841b82f6e" member_type: VOTER last_known_addr { host: "127.14.147.193" port: 33167 } } peers { permanent_uuid: "60d7fd130c0f46ea83489a52a291a07a" member_type: VOTER last_known_addr { host: "127.14.147.194" port: 44459 } } peers { permanent_uuid: "1144095f691a4cac85b7a4890ae1ae78" member_type: VOTER last_known_addr { host: "127.14.147.195" port: 35113 } }
I20260929 22:50:45.760787 22944 raft_consensus.cc:374] T 75afa0d8359a4343a08ea58b91f5709a P 1144095f691a4cac85b7a4890ae1ae78 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 22:50:45.760807 22944 raft_consensus.cc:729] T 75afa0d8359a4343a08ea58b91f5709a P 1144095f691a4cac85b7a4890ae1ae78 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1144095f691a4cac85b7a4890ae1ae78, State: Initialized, Role: FOLLOWER
I20260929 22:50:45.760850 22944 consensus_queue.cc:260] T 75afa0d8359a4343a08ea58b91f5709a P 1144095f691a4cac85b7a4890ae1ae78 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a46aea30b8a8431189d809e841b82f6e" member_type: VOTER last_known_addr { host: "127.14.147.193" port: 33167 } } peers { permanent_uuid: "60d7fd130c0f46ea83489a52a291a07a" member_type: VOTER last_known_addr { host: "127.14.147.194" port: 44459 } } peers { permanent_uuid: "1144095f691a4cac85b7a4890ae1ae78" member_type: VOTER last_known_addr { host: "127.14.147.195" port: 35113 } }
I20260929 22:50:45.760896 22946 tablet_bootstrap.cc:492] T 75afa0d8359a4343a08ea58b91f5709a P 60d7fd130c0f46ea83489a52a291a07a: No bootstrap required, opened a new log
I20260929 22:50:45.760941 22946 ts_tablet_manager.cc:1397] T 75afa0d8359a4343a08ea58b91f5709a P 60d7fd130c0f46ea83489a52a291a07a: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260929 22:50:45.760949 22944 ts_tablet_manager.cc:1428] T 75afa0d8359a4343a08ea58b91f5709a P 1144095f691a4cac85b7a4890ae1ae78: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260929 22:50:45.761055 22945 tablet_bootstrap.cc:492] T 75afa0d8359a4343a08ea58b91f5709a P a46aea30b8a8431189d809e841b82f6e: No bootstrap required, opened a new log
I20260929 22:50:45.761046 22946 raft_consensus.cc:348] T 75afa0d8359a4343a08ea58b91f5709a P 60d7fd130c0f46ea83489a52a291a07a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a46aea30b8a8431189d809e841b82f6e" member_type: VOTER last_known_addr { host: "127.14.147.193" port: 33167 } } peers { permanent_uuid: "60d7fd130c0f46ea83489a52a291a07a" member_type: VOTER last_known_addr { host: "127.14.147.194" port: 44459 } } peers { permanent_uuid: "1144095f691a4cac85b7a4890ae1ae78" member_type: VOTER last_known_addr { host: "127.14.147.195" port: 35113 } }
I20260929 22:50:45.761101 22945 ts_tablet_manager.cc:1397] T 75afa0d8359a4343a08ea58b91f5709a P a46aea30b8a8431189d809e841b82f6e: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260929 22:50:45.761102 22946 raft_consensus.cc:374] T 75afa0d8359a4343a08ea58b91f5709a P 60d7fd130c0f46ea83489a52a291a07a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 22:50:45.761121 22946 raft_consensus.cc:729] T 75afa0d8359a4343a08ea58b91f5709a P 60d7fd130c0f46ea83489a52a291a07a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 60d7fd130c0f46ea83489a52a291a07a, State: Initialized, Role: FOLLOWER
I20260929 22:50:45.761166 22946 consensus_queue.cc:260] T 75afa0d8359a4343a08ea58b91f5709a P 60d7fd130c0f46ea83489a52a291a07a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a46aea30b8a8431189d809e841b82f6e" member_type: VOTER last_known_addr { host: "127.14.147.193" port: 33167 } } peers { permanent_uuid: "60d7fd130c0f46ea83489a52a291a07a" member_type: VOTER last_known_addr { host: "127.14.147.194" port: 44459 } } peers { permanent_uuid: "1144095f691a4cac85b7a4890ae1ae78" member_type: VOTER last_known_addr { host: "127.14.147.195" port: 35113 } }
I20260929 22:50:45.761256 22946 ts_tablet_manager.cc:1428] T 75afa0d8359a4343a08ea58b91f5709a P 60d7fd130c0f46ea83489a52a291a07a: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260929 22:50:45.761233 22945 raft_consensus.cc:348] T 75afa0d8359a4343a08ea58b91f5709a P a46aea30b8a8431189d809e841b82f6e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a46aea30b8a8431189d809e841b82f6e" member_type: VOTER last_known_addr { host: "127.14.147.193" port: 33167 } } peers { permanent_uuid: "60d7fd130c0f46ea83489a52a291a07a" member_type: VOTER last_known_addr { host: "127.14.147.194" port: 44459 } } peers { permanent_uuid: "1144095f691a4cac85b7a4890ae1ae78" member_type: VOTER last_known_addr { host: "127.14.147.195" port: 35113 } }
I20260929 22:50:45.761291 22945 raft_consensus.cc:374] T 75afa0d8359a4343a08ea58b91f5709a P a46aea30b8a8431189d809e841b82f6e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 22:50:45.761314 22945 raft_consensus.cc:729] T 75afa0d8359a4343a08ea58b91f5709a P a46aea30b8a8431189d809e841b82f6e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a46aea30b8a8431189d809e841b82f6e, State: Initialized, Role: FOLLOWER
I20260929 22:50:45.761374 22945 consensus_queue.cc:260] T 75afa0d8359a4343a08ea58b91f5709a P a46aea30b8a8431189d809e841b82f6e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a46aea30b8a8431189d809e841b82f6e" member_type: VOTER last_known_addr { host: "127.14.147.193" port: 33167 } } peers { permanent_uuid: "60d7fd130c0f46ea83489a52a291a07a" member_type: VOTER last_known_addr { host: "127.14.147.194" port: 44459 } } peers { permanent_uuid: "1144095f691a4cac85b7a4890ae1ae78" member_type: VOTER last_known_addr { host: "127.14.147.195" port: 35113 } }
I20260929 22:50:45.761466 22945 ts_tablet_manager.cc:1428] T 75afa0d8359a4343a08ea58b91f5709a P a46aea30b8a8431189d809e841b82f6e: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260929 22:50:46.010581 22950 raft_consensus.cc:482] T 75afa0d8359a4343a08ea58b91f5709a P a46aea30b8a8431189d809e841b82f6e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 22:50:46.010677 22950 raft_consensus.cc:504] T 75afa0d8359a4343a08ea58b91f5709a P a46aea30b8a8431189d809e841b82f6e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a46aea30b8a8431189d809e841b82f6e" member_type: VOTER last_known_addr { host: "127.14.147.193" port: 33167 } } peers { permanent_uuid: "60d7fd130c0f46ea83489a52a291a07a" member_type: VOTER last_known_addr { host: "127.14.147.194" port: 44459 } } peers { permanent_uuid: "1144095f691a4cac85b7a4890ae1ae78" member_type: VOTER last_known_addr { host: "127.14.147.195" port: 35113 } }
I20260929 22:50:46.010984 22950 leader_election.cc:288] T 75afa0d8359a4343a08ea58b91f5709a P a46aea30b8a8431189d809e841b82f6e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 60d7fd130c0f46ea83489a52a291a07a (127.14.147.194:44459), 1144095f691a4cac85b7a4890ae1ae78 (127.14.147.195:35113)
I20260929 22:50:46.013793 22941 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:50:46.011244 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.147.193:37241 (local address 127.14.147.195:35113)
0929 22:50:46.011297 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0929 22:50:46.011299 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 22:50:46.011392 (+ 93us) server_negotiation.cc:407] Connection header received
0929 22:50:46.011473 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 22:50:46.011474 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 22:50:46.011484 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 22:50:46.011527 (+ 43us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 22:50:46.011659 (+ 132us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:50:46.012123 (+ 464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:50:46.013138 (+ 1015us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:50:46.013490 (+ 352us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:50:46.013549 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 22:50:46.013560 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0929 22:50:46.013682 (+ 122us) server_negotiation.cc:299] Negotiation successful
0929 22:50:46.013716 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260929 22:50:46.013801 22943 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:50:46.011291 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.147.193:40533 (local address 127.14.147.194:44459)
0929 22:50:46.011354 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0929 22:50:46.011356 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 22:50:46.011443 (+ 87us) server_negotiation.cc:407] Connection header received
0929 22:50:46.011519 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 22:50:46.011520 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 22:50:46.011532 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 22:50:46.011577 (+ 45us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 22:50:46.011711 (+ 134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:50:46.012160 (+ 449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:50:46.013133 (+ 973us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:50:46.013490 (+ 357us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:50:46.013549 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 22:50:46.013558 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0929 22:50:46.013696 (+ 138us) server_negotiation.cc:299] Negotiation successful
0929 22:50:46.013735 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260929 22:50:46.013815 23022 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:50:46.011169 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.147.195:35113 (local address 127.14.147.193:37241)
0929 22:50:46.011309 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0929 22:50:46.011320 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0929 22:50:46.011388 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 22:50:46.011543 (+ 155us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 22:50:46.011544 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 22:50:46.011546 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 22:50:46.011600 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 22:50:46.011602 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 22:50:46.012205 (+ 603us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 22:50:46.012206 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 22:50:46.013044 (+ 838us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 22:50:46.013047 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 22:50:46.013561 (+ 514us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 22:50:46.013562 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 22:50:46.013620 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 22:50:46.013624 (+ 4us) client_negotiation.cc:769] Sending connection context
0929 22:50:46.013667 (+ 43us) client_negotiation.cc:240] Negotiation successful
0929 22:50:46.013716 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":123,"mutex_wait_us":42,"thread_start_us":71,"threads_started":1}
I20260929 22:50:46.013835 23021 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:50:46.011172 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.147.194:44459 (local address 127.14.147.193:40533)
0929 22:50:46.011360 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0929 22:50:46.011369 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0929 22:50:46.011463 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 22:50:46.011586 (+ 123us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 22:50:46.011587 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 22:50:46.011589 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 22:50:46.011654 (+ 65us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 22:50:46.011656 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 22:50:46.012230 (+ 574us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 22:50:46.012231 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 22:50:46.013045 (+ 814us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 22:50:46.013047 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 22:50:46.013560 (+ 513us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 22:50:46.013561 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 22:50:46.013636 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 22:50:46.013640 (+ 4us) client_negotiation.cc:769] Sending connection context
0929 22:50:46.013683 (+ 43us) client_negotiation.cc:240] Negotiation successful
0929 22:50:46.013724 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":172,"mutex_wait_us":1,"thread_start_us":62,"threads_started":1}
I20260929 22:50:46.014103 22870 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "75afa0d8359a4343a08ea58b91f5709a" candidate_uuid: "a46aea30b8a8431189d809e841b82f6e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1144095f691a4cac85b7a4890ae1ae78" is_pre_election: true
I20260929 22:50:46.014195 22870 raft_consensus.cc:2454] T 75afa0d8359a4343a08ea58b91f5709a P 1144095f691a4cac85b7a4890ae1ae78 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a46aea30b8a8431189d809e841b82f6e in term 0.
I20260929 22:50:46.014233 22729 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "75afa0d8359a4343a08ea58b91f5709a" candidate_uuid: "a46aea30b8a8431189d809e841b82f6e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "60d7fd130c0f46ea83489a52a291a07a" is_pre_election: true
I20260929 22:50:46.014324 22729 raft_consensus.cc:2454] T 75afa0d8359a4343a08ea58b91f5709a P 60d7fd130c0f46ea83489a52a291a07a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a46aea30b8a8431189d809e841b82f6e in term 0.
I20260929 22:50:46.014364 22521 leader_election.cc:302] T 75afa0d8359a4343a08ea58b91f5709a P a46aea30b8a8431189d809e841b82f6e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1144095f691a4cac85b7a4890ae1ae78, a46aea30b8a8431189d809e841b82f6e; no voters:
I20260929 22:50:46.014495 22950 raft_consensus.cc:2781] T 75afa0d8359a4343a08ea58b91f5709a P a46aea30b8a8431189d809e841b82f6e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 22:50:46.014554 22950 raft_consensus.cc:482] T 75afa0d8359a4343a08ea58b91f5709a P a46aea30b8a8431189d809e841b82f6e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 22:50:46.014568 22950 raft_consensus.cc:3037] T 75afa0d8359a4343a08ea58b91f5709a P a46aea30b8a8431189d809e841b82f6e [term 0 FOLLOWER]: Advancing to term 1
I20260929 22:50:46.015259 22950 raft_consensus.cc:504] T 75afa0d8359a4343a08ea58b91f5709a P a46aea30b8a8431189d809e841b82f6e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a46aea30b8a8431189d809e841b82f6e" member_type: VOTER last_known_addr { host: "127.14.147.193" port: 33167 } } peers { permanent_uuid: "60d7fd130c0f46ea83489a52a291a07a" member_type: VOTER last_known_addr { host: "127.14.147.194" port: 44459 } } peers { permanent_uuid: "1144095f691a4cac85b7a4890ae1ae78" member_type: VOTER last_known_addr { host: "127.14.147.195" port: 35113 } }
I20260929 22:50:46.015383 22950 leader_election.cc:288] T 75afa0d8359a4343a08ea58b91f5709a P a46aea30b8a8431189d809e841b82f6e [CANDIDATE]: Term 1 election: Requested vote from peers 60d7fd130c0f46ea83489a52a291a07a (127.14.147.194:44459), 1144095f691a4cac85b7a4890ae1ae78 (127.14.147.195:35113)
I20260929 22:50:46.015542 22729 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "75afa0d8359a4343a08ea58b91f5709a" candidate_uuid: "a46aea30b8a8431189d809e841b82f6e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "60d7fd130c0f46ea83489a52a291a07a"
I20260929 22:50:46.015556 22870 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "75afa0d8359a4343a08ea58b91f5709a" candidate_uuid: "a46aea30b8a8431189d809e841b82f6e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1144095f691a4cac85b7a4890ae1ae78"
I20260929 22:50:46.015614 22729 raft_consensus.cc:3037] T 75afa0d8359a4343a08ea58b91f5709a P 60d7fd130c0f46ea83489a52a291a07a [term 0 FOLLOWER]: Advancing to term 1
I20260929 22:50:46.015630 22870 raft_consensus.cc:3037] T 75afa0d8359a4343a08ea58b91f5709a P 1144095f691a4cac85b7a4890ae1ae78 [term 0 FOLLOWER]: Advancing to term 1
I20260929 22:50:46.016135 22729 raft_consensus.cc:2454] T 75afa0d8359a4343a08ea58b91f5709a P 60d7fd130c0f46ea83489a52a291a07a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a46aea30b8a8431189d809e841b82f6e in term 1.
I20260929 22:50:46.016269 22870 raft_consensus.cc:2454] T 75afa0d8359a4343a08ea58b91f5709a P 1144095f691a4cac85b7a4890ae1ae78 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a46aea30b8a8431189d809e841b82f6e in term 1.
I20260929 22:50:46.016302 22522 leader_election.cc:302] T 75afa0d8359a4343a08ea58b91f5709a P a46aea30b8a8431189d809e841b82f6e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d7fd130c0f46ea83489a52a291a07a, a46aea30b8a8431189d809e841b82f6e; no voters:
I20260929 22:50:46.016373 22950 raft_consensus.cc:2781] T 75afa0d8359a4343a08ea58b91f5709a P a46aea30b8a8431189d809e841b82f6e [term 1 FOLLOWER]: Leader election won for term 1
I20260929 22:50:46.016489 22950 raft_consensus.cc:686] T 75afa0d8359a4343a08ea58b91f5709a P a46aea30b8a8431189d809e841b82f6e [term 1 LEADER]: Becoming Leader. State: Replica: a46aea30b8a8431189d809e841b82f6e, State: Running, Role: LEADER
I20260929 22:50:46.016558 22950 consensus_queue.cc:237] T 75afa0d8359a4343a08ea58b91f5709a P a46aea30b8a8431189d809e841b82f6e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a46aea30b8a8431189d809e841b82f6e" member_type: VOTER last_known_addr { host: "127.14.147.193" port: 33167 } } peers { permanent_uuid: "60d7fd130c0f46ea83489a52a291a07a" member_type: VOTER last_known_addr { host: "127.14.147.194" port: 44459 } } peers { permanent_uuid: "1144095f691a4cac85b7a4890ae1ae78" member_type: VOTER last_known_addr { host: "127.14.147.195" port: 35113 } }
I20260929 22:50:46.017285 22429 catalog_manager.cc:5495] T 75afa0d8359a4343a08ea58b91f5709a P a46aea30b8a8431189d809e841b82f6e reported cstate change: term changed from 0 to 1, leader changed from <none> to a46aea30b8a8431189d809e841b82f6e (127.14.147.193). New cstate: current_term: 1 leader_uuid: "a46aea30b8a8431189d809e841b82f6e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a46aea30b8a8431189d809e841b82f6e" member_type: VOTER last_known_addr { host: "127.14.147.193" port: 33167 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "60d7fd130c0f46ea83489a52a291a07a" member_type: VOTER last_known_addr { host: "127.14.147.194" port: 44459 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1144095f691a4cac85b7a4890ae1ae78" member_type: VOTER last_known_addr { host: "127.14.147.195" port: 35113 } health_report { overall_health: UNKNOWN } } }
I20260929 22:50:46.049028 23025 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:50:46.046725 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41286 (local address 127.14.147.193:33167)
0929 22:50:46.046890 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0929 22:50:46.046892 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 22:50:46.046898 (+ 6us) server_negotiation.cc:407] Connection header received
0929 22:50:46.046933 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 22:50:46.046934 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 22:50:46.046953 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 22:50:46.047018 (+ 65us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 22:50:46.047108 (+ 90us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:50:46.047678 (+ 570us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:50:46.048346 (+ 668us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:50:46.048561 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:50:46.048612 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 22:50:46.048711 (+ 99us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 22:50:46.048831 (+ 120us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 22:50:46.048880 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0929 22:50:46.048928 (+ 48us) server_negotiation.cc:299] Negotiation successful
0929 22:50:46.048944 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":146,"thread_start_us":117,"threads_started":1}
I20260929 22:50:46.053452 22941 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:50:46.051560 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.147.193:59413 (local address 127.14.147.195:35113)
0929 22:50:46.051609 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0929 22:50:46.051611 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 22:50:46.051621 (+ 10us) server_negotiation.cc:407] Connection header received
0929 22:50:46.051662 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 22:50:46.051663 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 22:50:46.051671 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 22:50:46.051711 (+ 40us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 22:50:46.051806 (+ 95us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:50:46.052258 (+ 452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:50:46.052949 (+ 691us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:50:46.053122 (+ 173us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:50:46.053162 (+ 40us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 22:50:46.053288 (+ 126us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 22:50:46.053311 (+ 23us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 22:50:46.053341 (+ 30us) server_negotiation.cc:1035] Waiting for connection context
0929 22:50:46.053377 (+ 36us) server_negotiation.cc:299] Negotiation successful
0929 22:50:46.053391 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260929 22:50:46.053488 23028 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:50:46.051376 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.147.195:35113 (local address 127.14.147.193:59413)
0929 22:50:46.051564 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0929 22:50:46.051575 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0929 22:50:46.051621 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 22:50:46.051718 (+ 97us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 22:50:46.051718 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 22:50:46.051720 (+ 2us) client_negotiation.cc:189] Negotiated authn=TOKEN
0929 22:50:46.051767 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 22:50:46.051768 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 22:50:46.052310 (+ 542us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 22:50:46.052310 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 22:50:46.052896 (+ 586us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 22:50:46.052897 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 22:50:46.053183 (+ 286us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 22:50:46.053183 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 22:50:46.053223 (+ 40us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 22:50:46.053236 (+ 13us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0929 22:50:46.053347 (+ 111us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0929 22:50:46.053348 (+ 1us) client_negotiation.cc:769] Sending connection context
0929 22:50:46.053375 (+ 27us) client_negotiation.cc:240] Negotiation successful
0929 22:50:46.053406 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":149,"thread_start_us":96,"threads_started":1}
I20260929 22:50:46.055543 22870 raft_consensus.cc:1261] T 75afa0d8359a4343a08ea58b91f5709a P 1144095f691a4cac85b7a4890ae1ae78 [term 1 FOLLOWER]: Refusing update from remote peer a46aea30b8a8431189d809e841b82f6e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 22:50:46.055562 22729 raft_consensus.cc:1261] T 75afa0d8359a4343a08ea58b91f5709a P 60d7fd130c0f46ea83489a52a291a07a [term 1 FOLLOWER]: Refusing update from remote peer a46aea30b8a8431189d809e841b82f6e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 22:50:46.055795 23023 consensus_queue.cc:1035] T 75afa0d8359a4343a08ea58b91f5709a P a46aea30b8a8431189d809e841b82f6e [LEADER]: Connected to new peer: Peer: permanent_uuid: "60d7fd130c0f46ea83489a52a291a07a" member_type: VOTER last_known_addr { host: "127.14.147.194" port: 44459 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 22:50:46.055927 22950 consensus_queue.cc:1035] T 75afa0d8359a4343a08ea58b91f5709a P a46aea30b8a8431189d809e841b82f6e [LEADER]: Connected to new peer: Peer: permanent_uuid: "1144095f691a4cac85b7a4890ae1ae78" member_type: VOTER last_known_addr { host: "127.14.147.195" port: 35113 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 22:50:46.062275 23025 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:50:46.060003 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.147.195:59721 (local address 127.14.147.193:33167)
0929 22:50:46.060040 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0929 22:50:46.060042 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 22:50:46.060050 (+ 8us) server_negotiation.cc:407] Connection header received
0929 22:50:46.060111 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 22:50:46.060112 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 22:50:46.060122 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 22:50:46.060173 (+ 51us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 22:50:46.060277 (+ 104us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:50:46.060737 (+ 460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:50:46.061495 (+ 758us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:50:46.061782 (+ 287us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:50:46.061861 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 22:50:46.062003 (+ 142us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 22:50:46.062053 (+ 50us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 22:50:46.062097 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0929 22:50:46.062180 (+ 83us) server_negotiation.cc:299] Negotiation successful
0929 22:50:46.062201 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260929 22:50:46.062283 22922 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:50:46.059974 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.147.193:33167 (local address 127.14.147.195:59721)
0929 22:50:46.060017 (+ 43us) negotiation.cc:107] Waiting for socket to connect
0929 22:50:46.060028 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0929 22:50:46.060062 (+ 34us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 22:50:46.060186 (+ 124us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 22:50:46.060187 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 22:50:46.060188 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0929 22:50:46.060235 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 22:50:46.060237 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 22:50:46.060813 (+ 576us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 22:50:46.060814 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 22:50:46.061415 (+ 601us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 22:50:46.061417 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 22:50:46.061862 (+ 445us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 22:50:46.061863 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 22:50:46.061931 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 22:50:46.061937 (+ 6us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0929 22:50:46.062124 (+ 187us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0929 22:50:46.062126 (+ 2us) client_negotiation.cc:769] Sending connection context
0929 22:50:46.062163 (+ 37us) client_negotiation.cc:240] Negotiation successful
0929 22:50:46.062216 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":25}
I20260929 22:50:46.068522 22428 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46540:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 22:50:46.068639 22428 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46540:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 22:50:46.069286 22428 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 7c99fb093ec947debdad2c01a7fd8027: Sending DeleteTablet for 3 replicas of tablet 75afa0d8359a4343a08ea58b91f5709a
I20260929 22:50:46.069621 22709 tablet_service.cc:1514] Processing DeleteTablet for tablet 75afa0d8359a4343a08ea58b91f5709a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 22:50:46 UTC) from {username='oryx', principal='oryx/127.14.147.254@KRBTEST.COM'} at 127.0.0.1:46792
I20260929 22:50:46.069607 22568 tablet_service.cc:1514] Processing DeleteTablet for tablet 75afa0d8359a4343a08ea58b91f5709a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 22:50:46 UTC) from {username='oryx', principal='oryx/127.14.147.254@KRBTEST.COM'} at 127.0.0.1:41272
I20260929 22:50:46.069730 22969 tablet_replica.cc:330] T 75afa0d8359a4343a08ea58b91f5709a P a46aea30b8a8431189d809e841b82f6e: stopping tablet replica
I20260929 22:50:46.069736 22968 tablet_replica.cc:330] T 75afa0d8359a4343a08ea58b91f5709a P 60d7fd130c0f46ea83489a52a291a07a: stopping tablet replica
I20260929 22:50:46.069772 22969 raft_consensus.cc:2229] T 75afa0d8359a4343a08ea58b91f5709a P a46aea30b8a8431189d809e841b82f6e [term 1 LEADER]: Raft consensus shutting down.
I20260929 22:50:46.069811 22968 raft_consensus.cc:2229] T 75afa0d8359a4343a08ea58b91f5709a P 60d7fd130c0f46ea83489a52a291a07a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 22:50:46.069847 22968 raft_consensus.cc:2258] T 75afa0d8359a4343a08ea58b91f5709a P 60d7fd130c0f46ea83489a52a291a07a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 22:50:46.069861 22969 raft_consensus.cc:2258] T 75afa0d8359a4343a08ea58b91f5709a P a46aea30b8a8431189d809e841b82f6e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 22:50:46.070145 22968 ts_tablet_manager.cc:1905] T 75afa0d8359a4343a08ea58b91f5709a P 60d7fd130c0f46ea83489a52a291a07a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 22:50:46.070258 22969 ts_tablet_manager.cc:1905] T 75afa0d8359a4343a08ea58b91f5709a P a46aea30b8a8431189d809e841b82f6e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 22:50:46.070379 14927 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhWOCLk/build/release/bin/kudu with pid 22503
I20260929 22:50:46.071887 22968 ts_tablet_manager.cc:1918] T 75afa0d8359a4343a08ea58b91f5709a P 60d7fd130c0f46ea83489a52a291a07a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260929 22:50:46.071930 22968 log.cc:1196] T 75afa0d8359a4343a08ea58b91f5709a P 60d7fd130c0f46ea83489a52a291a07a: Deleting WAL directory at /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790722213368738-14927-0/minicluster-data/ts-1/wal/wals/75afa0d8359a4343a08ea58b91f5709a
I20260929 22:50:46.072067 22968 ts_tablet_manager.cc:1939] T 75afa0d8359a4343a08ea58b91f5709a P 60d7fd130c0f46ea83489a52a291a07a: Deleting consensus metadata
I20260929 22:50:46.072335 22417 catalog_manager.cc:4841] TS 60d7fd130c0f46ea83489a52a291a07a (127.14.147.194:44459): tablet 75afa0d8359a4343a08ea58b91f5709a (table test-table [id=d3d8e98e5f3648fc992b480b878f7b48]) successfully deleted
I20260929 22:50:46.072515 22835 tablet_service.cc:1514] Processing DeleteTablet for tablet 75afa0d8359a4343a08ea58b91f5709a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 22:50:46 UTC) from {username='oryx', principal='oryx/127.14.147.254@KRBTEST.COM'} at 127.0.0.1:49068
I20260929 22:50:46.072662 22967 tablet_replica.cc:330] T 75afa0d8359a4343a08ea58b91f5709a P 1144095f691a4cac85b7a4890ae1ae78: stopping tablet replica
I20260929 22:50:46.072721 22967 raft_consensus.cc:2229] T 75afa0d8359a4343a08ea58b91f5709a P 1144095f691a4cac85b7a4890ae1ae78 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 22:50:46.072764 22967 raft_consensus.cc:2258] T 75afa0d8359a4343a08ea58b91f5709a P 1144095f691a4cac85b7a4890ae1ae78 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 22:50:46.073441 22967 ts_tablet_manager.cc:1905] T 75afa0d8359a4343a08ea58b91f5709a P 1144095f691a4cac85b7a4890ae1ae78: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 22:50:46.074573 22967 ts_tablet_manager.cc:1918] T 75afa0d8359a4343a08ea58b91f5709a P 1144095f691a4cac85b7a4890ae1ae78: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260929 22:50:46.074621 22967 log.cc:1196] T 75afa0d8359a4343a08ea58b91f5709a P 1144095f691a4cac85b7a4890ae1ae78: Deleting WAL directory at /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790722213368738-14927-0/minicluster-data/ts-2/wal/wals/75afa0d8359a4343a08ea58b91f5709a
I20260929 22:50:46.074792 22967 ts_tablet_manager.cc:1939] T 75afa0d8359a4343a08ea58b91f5709a P 1144095f691a4cac85b7a4890ae1ae78: Deleting consensus metadata
I20260929 22:50:46.075103 22416 catalog_manager.cc:4841] TS 1144095f691a4cac85b7a4890ae1ae78 (127.14.147.195:35113): tablet 75afa0d8359a4343a08ea58b91f5709a (table test-table [id=d3d8e98e5f3648fc992b480b878f7b48]) successfully deleted
W20260929 22:50:46.076061 22417 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.14.147.193:33167 (error 108)
I20260929 22:50:46.076294 14927 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhWOCLk/build/release/bin/kudu with pid 22643
I20260929 22:50:46.079664 23035 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:50:46.076398 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.147.193:33167 (local address 127.0.0.1:41288)
0929 22:50:46.079549 (+ 3151us) negotiation.cc:107] Waiting for socket to connect
0929 22:50:46.079622 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.147.193:33167: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":3130,"thread_start_us":40,"threads_started":1}
W20260929 22:50:46.079814 22417 catalog_manager.cc:4584] TS a46aea30b8a8431189d809e841b82f6e (127.14.147.193:33167): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 75afa0d8359a4343a08ea58b91f5709a: Network error: Client connection negotiation failed: client connection to 127.14.147.193:33167: connect: Connection refused (error 111)
I20260929 22:50:46.082507 14927 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhWOCLk/build/release/bin/kudu with pid 22784
I20260929 22:50:46.088312 14927 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhWOCLk/build/release/bin/kudu with pid 22398
2026-09-29T22:50:46Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipal (2513 ms)
[ RUN ] SecurityITest.TestNonExistentPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790722213368738-14927-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 22:50:46 dist-test-slave-98mk krb5kdc[23038](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 22:50:46 dist-test-slave-98mk krb5kdc[23038](info): set up 2 sockets
Sep 29 22:50:46 dist-test-slave-98mk krb5kdc[23038](info): commencing operation
krb5kdc: starting...
W20260929 22:50:47.036399 14927 mini_kdc.cc:121] Time spent starting KDC: real 0.928s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790722213368738-14927-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790722213368738-14927-0/krb5kdc/test-user.keytab.
Sep 29 22:50:47 dist-test-slave-98mk krb5kdc[23038](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790722247, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T22:50:47Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T22:50:47Z Disabled control of system clock
WARNING: no policy specified for kudu/127.14.147.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.147.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.147.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790722213368738-14927-0/krb5kdc/kudu_127.14.147.254.keytab.
Entry for principal kudu/127.14.147.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790722213368738-14927-0/krb5kdc/kudu_127.14.147.254.keytab.
I20260929 22:50:47.145951 14927 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhWOCLk/build/release/bin/kudu
/tmp/dist-test-taskhWOCLk/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790722213368738-14927-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790722213368738-14927-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790722213368738-14927-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790722213368738-14927-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.147.254:39291
--webserver_interface=127.14.147.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790722213368738-14927-0/krb5kdc/kudu_127.14.147.254.keytab
--principal=kudu/127.14.147.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.147.212:45359
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.147.254:39291
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--principal=oryx with env {KRB5CCNAME=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790722213368738-14927-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790722213368738-14927-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790722213368738-14927-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 22:50:47.226118 23051 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 22:50:47.226372 23051 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 22:50:47.226397 23051 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 22:50:47.227727 23051 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 22:50:47.227777 23051 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 22:50:47.227792 23051 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 22:50:47.227803 23051 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 22:50:47.227814 23051 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 22:50:47.229522 23051 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.147.212:45359
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790722213368738-14927-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790722213368738-14927-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.14.147.254:39291
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790722213368738-14927-0/krb5kdc/kudu_127.14.147.254.keytab
--principal=oryx
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.147.254:39291
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790722213368738-14927-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.14.147.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23051
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790722213368738-14927-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 21:58:54 UTC on ccycloud.quasar-xguoti.root.comops.site
build id 4681390
I20260929 22:50:47.229786 23051 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 22:50:47.230003 23051 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 22:50:47.234839 23057 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 22:50:47.234859 23056 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 22:50:47.234845 23059 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 22:50:47.234987 23051 server_base.cc:1034] running on GCE node
I20260929 22:50:47.235237 23051 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 22:50:47.235555 23051 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 22:50:47.236716 23051 hybrid_clock.cc:648] HybridClock initialized: now 1790722247236693 us; error 32 us; skew 500 ppm
Runtime error: kinit failed: unable to login from keytab: Keytab contains no suitable keys for oryx@KRBTEST.COM
2026-09-29T22:50:47Z chronyd exiting
[ OK ] SecurityITest.TestNonExistentPrincipal (1155 ms)
[ RUN ] SecurityITest.TestMismatchingPrincipals
Loading random data
Initializing database '/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790722213368738-14927-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 22:50:47 dist-test-slave-98mk krb5kdc[23064](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 22:50:47 dist-test-slave-98mk krb5kdc[23064](info): set up 2 sockets
Sep 29 22:50:47 dist-test-slave-98mk krb5kdc[23064](info): commencing operation
krb5kdc: starting...
W20260929 22:50:48.189680 14927 mini_kdc.cc:121] Time spent starting KDC: real 0.926s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790722213368738-14927-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790722213368738-14927-0/krb5kdc/test-user.keytab.
Sep 29 22:50:48 dist-test-slave-98mk krb5kdc[23064](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790722248, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T22:50:48Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T22:50:48Z Disabled control of system clock
WARNING: no policy specified for kudu/127.14.147.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.147.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.147.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790722213368738-14927-0/krb5kdc/kudu_127.14.147.254.keytab.
Entry for principal kudu/127.14.147.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790722213368738-14927-0/krb5kdc/kudu_127.14.147.254.keytab.
I20260929 22:50:48.310159 14927 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhWOCLk/build/release/bin/kudu
/tmp/dist-test-taskhWOCLk/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790722213368738-14927-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790722213368738-14927-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790722213368738-14927-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790722213368738-14927-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.147.254:40759
--webserver_interface=127.14.147.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790722213368738-14927-0/krb5kdc/kudu_127.14.147.254.keytab
--principal=kudu/127.14.147.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.147.212:44407
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.147.254:40759
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790722213368738-14927-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790722213368738-14927-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790722213368738-14927-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 22:50:48.390017 23077 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 22:50:48.390259 23077 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 22:50:48.390292 23077 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 22:50:48.391563 23077 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 22:50:48.391608 23077 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 22:50:48.391628 23077 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 22:50:48.391644 23077 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 22:50:48.391667 23077 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 22:50:48.393280 23077 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.147.212:44407
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790722213368738-14927-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790722213368738-14927-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.14.147.254:40759
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790722213368738-14927-0/krb5kdc/kudu_127.14.147.254.keytab
--principal=kudu/127.14.147.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.147.254:40759
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790722213368738-14927-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.14.147.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23077
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790722213368738-14927-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 21:58:54 UTC on ccycloud.quasar-xguoti.root.comops.site
build id 4681390
I20260929 22:50:48.393548 23077 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 22:50:48.393765 23077 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 22:50:48.398892 23085 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 22:50:48.398916 23083 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 22:50:48.398901 23082 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 22:50:48.399212 23077 server_base.cc:1034] running on GCE node
I20260929 22:50:48.399493 23077 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 22:50:48.399803 23077 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 22:50:48.400957 23077 hybrid_clock.cc:648] HybridClock initialized: now 1790722248400937 us; error 36 us; skew 500 ppm
Sep 29 22:50:48 dist-test-slave-98mk krb5kdc[23064](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790722248, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.147.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 22:50:48.403683 23077 init.cc:377] Logged in from keytab as kudu/127.14.147.254@KRBTEST.COM (short username kudu)
I20260929 22:50:48.404201 23077 webserver.cc:460] Webserver started at http://127.14.147.254:34221/ using document root <none> and password file <none>
I20260929 22:50:48.404436 23077 fs_manager.cc:359] Metadata directory not provided
I20260929 22:50:48.404487 23077 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 22:50:48.404623 23077 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 22:50:48.405709 23077 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790722213368738-14927-0/minicluster-data/master-0/data/instance:
uuid: "316eee16b8584fbdb2deae855af51c1e"
format_stamp: "Formatted at 2026-09-29 22:50:48 on dist-test-slave-98mk"
server_key: "0de1e1db88ce4c47b4a95a07f051bdb8"
server_key_iv: "630657ccbddaaa82d241a91cfd869948"
server_key_version: "encryptionkey@0"
I20260929 22:50:48.406064 23077 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790722213368738-14927-0/minicluster-data/master-0/wal/instance:
uuid: "316eee16b8584fbdb2deae855af51c1e"
format_stamp: "Formatted at 2026-09-29 22:50:48 on dist-test-slave-98mk"
server_key: "0de1e1db88ce4c47b4a95a07f051bdb8"
server_key_iv: "630657ccbddaaa82d241a91cfd869948"
server_key_version: "encryptionkey@0"
I20260929 22:50:48.407490 23077 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260929 22:50:48.408308 23092 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 22:50:48.408484 23077 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260929 22:50:48.408573 23077 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790722213368738-14927-0/minicluster-data/master-0/data,/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790722213368738-14927-0/minicluster-data/master-0/wal
uuid: "316eee16b8584fbdb2deae855af51c1e"
format_stamp: "Formatted at 2026-09-29 22:50:48 on dist-test-slave-98mk"
server_key: "0de1e1db88ce4c47b4a95a07f051bdb8"
server_key_iv: "630657ccbddaaa82d241a91cfd869948"
server_key_version: "encryptionkey@0"
I20260929 22:50:48.408653 23077 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790722213368738-14927-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790722213368738-14927-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790722213368738-14927-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 22:50:48.419275 23077 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 22:50:48.421348 23077 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 22:50:48.421506 23077 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 22:50:48.425433 23077 rpc_server.cc:307] RPC server started. Bound to: 127.14.147.254:40759
I20260929 22:50:48.425475 23144 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.147.254:40759 every 8 connection(s)
I20260929 22:50:48.425822 23077 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790722213368738-14927-0/minicluster-data/master-0/data/info.pb
I20260929 22:50:48.426744 23145 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 22:50:48.429071 23145 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 316eee16b8584fbdb2deae855af51c1e: Bootstrap starting.
I20260929 22:50:48.429664 23145 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 316eee16b8584fbdb2deae855af51c1e: Neither blocks nor log segments found. Creating new log.
I20260929 22:50:48.429951 23145 log.cc:824] T 00000000000000000000000000000000 P 316eee16b8584fbdb2deae855af51c1e: Log is configured to *not* fsync() on all Append() calls
I20260929 22:50:48.430661 23145 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 316eee16b8584fbdb2deae855af51c1e: No bootstrap required, opened a new log
I20260929 22:50:48.432366 23145 raft_consensus.cc:348] T 00000000000000000000000000000000 P 316eee16b8584fbdb2deae855af51c1e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "316eee16b8584fbdb2deae855af51c1e" member_type: VOTER last_known_addr { host: "127.14.147.254" port: 40759 } }
I20260929 22:50:48.432515 23145 raft_consensus.cc:374] T 00000000000000000000000000000000 P 316eee16b8584fbdb2deae855af51c1e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 22:50:48.432536 23145 raft_consensus.cc:729] T 00000000000000000000000000000000 P 316eee16b8584fbdb2deae855af51c1e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 316eee16b8584fbdb2deae855af51c1e, State: Initialized, Role: FOLLOWER
I20260929 22:50:48.432674 23145 consensus_queue.cc:260] T 00000000000000000000000000000000 P 316eee16b8584fbdb2deae855af51c1e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "316eee16b8584fbdb2deae855af51c1e" member_type: VOTER last_known_addr { host: "127.14.147.254" port: 40759 } }
I20260929 22:50:48.432744 23145 raft_consensus.cc:388] T 00000000000000000000000000000000 P 316eee16b8584fbdb2deae855af51c1e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 22:50:48.432768 23145 raft_consensus.cc:482] T 00000000000000000000000000000000 P 316eee16b8584fbdb2deae855af51c1e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 22:50:48.432814 23145 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 316eee16b8584fbdb2deae855af51c1e [term 0 FOLLOWER]: Advancing to term 1
I20260929 22:50:48.433382 23145 raft_consensus.cc:504] T 00000000000000000000000000000000 P 316eee16b8584fbdb2deae855af51c1e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "316eee16b8584fbdb2deae855af51c1e" member_type: VOTER last_known_addr { host: "127.14.147.254" port: 40759 } }
I20260929 22:50:48.433472 23145 leader_election.cc:302] T 00000000000000000000000000000000 P 316eee16b8584fbdb2deae855af51c1e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 316eee16b8584fbdb2deae855af51c1e; no voters:
I20260929 22:50:48.433624 23145 leader_election.cc:288] T 00000000000000000000000000000000 P 316eee16b8584fbdb2deae855af51c1e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 22:50:48.433681 23148 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 316eee16b8584fbdb2deae855af51c1e [term 1 FOLLOWER]: Leader election won for term 1
I20260929 22:50:48.433874 23145 sys_catalog.cc:564] T 00000000000000000000000000000000 P 316eee16b8584fbdb2deae855af51c1e [sys.catalog]: configured and running, proceeding with master startup.
I20260929 22:50:48.433899 23148 raft_consensus.cc:686] T 00000000000000000000000000000000 P 316eee16b8584fbdb2deae855af51c1e [term 1 LEADER]: Becoming Leader. State: Replica: 316eee16b8584fbdb2deae855af51c1e, State: Running, Role: LEADER
I20260929 22:50:48.433988 23148 consensus_queue.cc:237] T 00000000000000000000000000000000 P 316eee16b8584fbdb2deae855af51c1e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "316eee16b8584fbdb2deae855af51c1e" member_type: VOTER last_known_addr { host: "127.14.147.254" port: 40759 } }
I20260929 22:50:48.434355 23150 sys_catalog.cc:455] T 00000000000000000000000000000000 P 316eee16b8584fbdb2deae855af51c1e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 316eee16b8584fbdb2deae855af51c1e. Latest consensus state: current_term: 1 leader_uuid: "316eee16b8584fbdb2deae855af51c1e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "316eee16b8584fbdb2deae855af51c1e" member_type: VOTER last_known_addr { host: "127.14.147.254" port: 40759 } } }
I20260929 22:50:48.434382 23149 sys_catalog.cc:455] T 00000000000000000000000000000000 P 316eee16b8584fbdb2deae855af51c1e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "316eee16b8584fbdb2deae855af51c1e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "316eee16b8584fbdb2deae855af51c1e" member_type: VOTER last_known_addr { host: "127.14.147.254" port: 40759 } } }
I20260929 22:50:48.434468 23149 sys_catalog.cc:458] T 00000000000000000000000000000000 P 316eee16b8584fbdb2deae855af51c1e [sys.catalog]: This master's current role is: LEADER
I20260929 22:50:48.434458 23150 sys_catalog.cc:458] T 00000000000000000000000000000000 P 316eee16b8584fbdb2deae855af51c1e [sys.catalog]: This master's current role is: LEADER
I20260929 22:50:48.434682 23152 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 22:50:48.435119 14927 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhWOCLk/build/release/bin/kudu as pid 23077
I20260929 22:50:48.435233 14927 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790722213368738-14927-0/minicluster-data/master-0/wal/instance
I20260929 22:50:48.435230 23152 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 22:50:48.435405 14927 external_mini_cluster.cc:1428] Setting key 27cbcbf1a2e4666d9e83702dda7b9792
I20260929 22:50:48.436970 23152 catalog_manager.cc:1347] Generated new cluster ID: 80a35ce9412b4812b44420efd2b010e4
I20260929 22:50:48.437019 23152 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 29 22:50:48 dist-test-slave-98mk krb5kdc[23064](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790722248, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.147.254@KRBTEST.COM
I20260929 22:50:48.443639 23152 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 22:50:48.444283 23152 catalog_manager.cc:1504] Loading token signing keys...
I20260929 22:50:48.447510 23167 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:50:48.435923 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48048 (local address 127.14.147.254:40759)
0929 22:50:48.436303 (+ 380us) server_negotiation.cc:206] Beginning negotiation
0929 22:50:48.436313 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 22:50:48.436341 (+ 28us) server_negotiation.cc:407] Connection header received
0929 22:50:48.436469 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 22:50:48.436476 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 22:50:48.436538 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 22:50:48.436673 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0929 22:50:48.437410 (+ 737us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:50:48.438881 (+ 1471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:50:48.439956 (+ 1075us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:50:48.440637 (+ 681us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:50:48.440767 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 22:50:48.443618 (+ 2851us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 22:50:48.443624 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 22:50:48.443624 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 22:50:48.443637 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0929 22:50:48.445691 (+ 2054us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 22:50:48.446097 (+ 406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 22:50:48.446100 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 22:50:48.446101 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 22:50:48.446218 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 22:50:48.446394 (+ 176us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 22:50:48.446395 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 22:50:48.446395 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 22:50:48.446692 (+ 297us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 22:50:48.446862 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0929 22:50:48.446936 (+ 74us) server_negotiation.cc:299] Negotiation successful
0929 22:50:48.447219 (+ 283us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":329,"thread_start_us":67,"threads_started":1}
I20260929 22:50:48.448179 23152 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 316eee16b8584fbdb2deae855af51c1e: Generated new TSK 0
I20260929 22:50:48.448356 23152 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.14.147.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.147.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.147.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790722213368738-14927-0/krb5kdc/kudu_127.14.147.193.keytab.
Entry for principal kudu/127.14.147.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790722213368738-14927-0/krb5kdc/kudu_127.14.147.193.keytab.
I20260929 22:50:48.479794 14927 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhWOCLk/build/release/bin/kudu
/tmp/dist-test-taskhWOCLk/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790722213368738-14927-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790722213368738-14927-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790722213368738-14927-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790722213368738-14927-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.147.193:0
--local_ip_for_outbound_sockets=127.14.147.193
--webserver_interface=127.14.147.193
--webserver_port=0
--tserver_master_addrs=127.14.147.254:40759
--keytab_file=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790722213368738-14927-0/krb5kdc/kudu_127.14.147.193.keytab
--principal=kudu/127.14.147.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.147.212:44407
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790722213368738-14927-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790722213368738-14927-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790722213368738-14927-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 22:50:48.582871 23171 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 22:50:48.583088 23171 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 22:50:48.583122 23171 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 22:50:48.584396 23171 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 22:50:48.584444 23171 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 22:50:48.584482 23171 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.147.193
I20260929 22:50:48.585973 23171 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.147.212:44407
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790722213368738-14927-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790722213368738-14927-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790722213368738-14927-0/krb5kdc/kudu_127.14.147.193.keytab
--principal=kudu/127.14.147.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.147.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790722213368738-14927-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.14.147.193
--webserver_port=0
--tserver_master_addrs=127.14.147.254:40759
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23171
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.147.193
--log_dir=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790722213368738-14927-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 21:58:54 UTC on ccycloud.quasar-xguoti.root.comops.site
build id 4681390
I20260929 22:50:48.586221 23171 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 22:50:48.586443 23171 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 22:50:48.592125 23179 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 22:50:48.592159 23177 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 22:50:48.592159 23176 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 22:50:48.592445 23171 server_base.cc:1034] running on GCE node
I20260929 22:50:48.592682 23171 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 22:50:48.592941 23171 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 22:50:48.594101 23171 hybrid_clock.cc:648] HybridClock initialized: now 1790722248594086 us; error 30 us; skew 500 ppm
Sep 29 22:50:48 dist-test-slave-98mk krb5kdc[23064](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790722248, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.147.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 22:50:48.599759 23171 init.cc:377] Logged in from keytab as kudu/127.14.147.193@KRBTEST.COM (short username kudu)
I20260929 22:50:48.600219 23171 webserver.cc:460] Webserver started at http://127.14.147.193:35881/ using document root <none> and password file <none>
I20260929 22:50:48.600448 23171 fs_manager.cc:359] Metadata directory not provided
I20260929 22:50:48.600493 23171 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 22:50:48.600623 23171 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 22:50:48.601670 23171 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790722213368738-14927-0/minicluster-data/ts-0/data/instance:
uuid: "5da67f43f97f46669eaf53392a39f370"
format_stamp: "Formatted at 2026-09-29 22:50:48 on dist-test-slave-98mk"
server_key: "d864017c8b75c5a0109a10d6b4979e83"
server_key_iv: "19909f39ec587f3ce111f2dc4f169eb5"
server_key_version: "encryptionkey@0"
I20260929 22:50:48.601990 23171 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790722213368738-14927-0/minicluster-data/ts-0/wal/instance:
uuid: "5da67f43f97f46669eaf53392a39f370"
format_stamp: "Formatted at 2026-09-29 22:50:48 on dist-test-slave-98mk"
server_key: "d864017c8b75c5a0109a10d6b4979e83"
server_key_iv: "19909f39ec587f3ce111f2dc4f169eb5"
server_key_version: "encryptionkey@0"
I20260929 22:50:48.603372 23171 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260929 22:50:48.604333 23186 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 22:50:48.604542 23171 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260929 22:50:48.604628 23171 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790722213368738-14927-0/minicluster-data/ts-0/data,/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790722213368738-14927-0/minicluster-data/ts-0/wal
uuid: "5da67f43f97f46669eaf53392a39f370"
format_stamp: "Formatted at 2026-09-29 22:50:48 on dist-test-slave-98mk"
server_key: "d864017c8b75c5a0109a10d6b4979e83"
server_key_iv: "19909f39ec587f3ce111f2dc4f169eb5"
server_key_version: "encryptionkey@0"
I20260929 22:50:48.604705 23171 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790722213368738-14927-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790722213368738-14927-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790722213368738-14927-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 22:50:48.633152 23171 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 22:50:48.635232 23171 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 22:50:48.635365 23171 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 22:50:48.635600 23171 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 22:50:48.635936 23171 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 22:50:48.635974 23171 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 22:50:48.636004 23171 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 22:50:48.636029 23171 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 22:50:48.641400 23171 rpc_server.cc:307] RPC server started. Bound to: 127.14.147.193:36425
I20260929 22:50:48.641471 23300 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.147.193:36425 every 8 connection(s)
I20260929 22:50:48.641760 23171 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790722213368738-14927-0/minicluster-data/ts-0/data/info.pb
I20260929 22:50:48.646031 14927 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhWOCLk/build/release/bin/kudu as pid 23171
I20260929 22:50:48.646134 14927 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790722213368738-14927-0/minicluster-data/ts-0/wal/instance
I20260929 22:50:48.646301 14927 external_mini_cluster.cc:1428] Setting key f24e2b56a15fef8a3ab03afc9ebdb4a9
Sep 29 22:50:48 dist-test-slave-98mk krb5kdc[23064](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790722248, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.147.193@KRBTEST.COM for kudu/127.14.147.254@KRBTEST.COM
I20260929 22:50:48.651458 23167 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:50:48.642475 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.147.193:41329 (local address 127.14.147.254:40759)
0929 22:50:48.642591 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0929 22:50:48.642593 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 22:50:48.642757 (+ 164us) server_negotiation.cc:407] Connection header received
0929 22:50:48.643275 (+ 518us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 22:50:48.643276 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 22:50:48.643288 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 22:50:48.643341 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0929 22:50:48.644291 (+ 950us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:50:48.644806 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:50:48.645783 (+ 977us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:50:48.645998 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:50:48.646054 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 22:50:48.648293 (+ 2239us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 22:50:48.648295 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 22:50:48.648295 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 22:50:48.648304 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0929 22:50:48.649823 (+ 1519us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 22:50:48.650460 (+ 637us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 22:50:48.650462 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 22:50:48.650463 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 22:50:48.650531 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 22:50:48.650765 (+ 234us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 22:50:48.650766 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 22:50:48.650766 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 22:50:48.650883 (+ 117us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 22:50:48.650942 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0929 22:50:48.651209 (+ 267us) server_negotiation.cc:299] Negotiation successful
0929 22:50:48.651318 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260929 22:50:48.651589 23303 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:50:48.642428 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.147.254:40759 (local address 127.14.147.193:41329)
0929 22:50:48.642632 (+ 204us) negotiation.cc:107] Waiting for socket to connect
0929 22:50:48.642670 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0929 22:50:48.643170 (+ 500us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 22:50:48.643407 (+ 237us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 22:50:48.643410 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 22:50:48.643991 (+ 581us) client_negotiation.cc:189] Negotiated authn=SASL
0929 22:50:48.644185 (+ 194us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 22:50:48.644190 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 22:50:48.644901 (+ 711us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 22:50:48.644902 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 22:50:48.645698 (+ 796us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 22:50:48.645700 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 22:50:48.646070 (+ 370us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 22:50:48.646071 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 22:50:48.646201 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 22:50:48.646749 (+ 548us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 22:50:48.646759 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0929 22:50:48.648202 (+ 1443us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 22:50:48.650059 (+ 1857us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 22:50:48.650068 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 22:50:48.650071 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 22:50:48.650368 (+ 297us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 22:50:48.650590 (+ 222us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 22:50:48.650590 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 22:50:48.650591 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 22:50:48.650707 (+ 116us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 22:50:48.650938 (+ 231us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 22:50:48.650944 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 22:50:48.651089 (+ 145us) client_negotiation.cc:769] Sending connection context
0929 22:50:48.651179 (+ 90us) client_negotiation.cc:240] Negotiation successful
0929 22:50:48.651342 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":158,"thread_start_us":82,"threads_started":1}
I20260929 22:50:48.652012 23301 heartbeater.cc:347] Connected to a master server at 127.14.147.254:40759
I20260929 22:50:48.652112 23301 heartbeater.cc:464] Registering TS with master...
I20260929 22:50:48.652282 23301 heartbeater.cc:511] Master 127.14.147.254:40759 requested a full tablet report, sending...
I20260929 22:50:48.652760 23109 ts_manager.cc:194] Registered new tserver with Master: 5da67f43f97f46669eaf53392a39f370 (127.14.147.193:36425)
I20260929 22:50:48.653553 23109 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.147.193@KRBTEST.COM'} at 127.14.147.193:41329
WARNING: no policy specified for kudu/127.14.147.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.147.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.147.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790722213368738-14927-0/krb5kdc/kudu_127.14.147.194.keytab.
Entry for principal kudu/127.14.147.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790722213368738-14927-0/krb5kdc/kudu_127.14.147.194.keytab.
I20260929 22:50:48.668507 14927 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhWOCLk/build/release/bin/kudu
/tmp/dist-test-taskhWOCLk/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790722213368738-14927-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790722213368738-14927-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790722213368738-14927-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790722213368738-14927-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.147.194:0
--local_ip_for_outbound_sockets=127.14.147.194
--webserver_interface=127.14.147.194
--webserver_port=0
--tserver_master_addrs=127.14.147.254:40759
--keytab_file=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790722213368738-14927-0/krb5kdc/kudu_127.14.147.194.keytab
--principal=kudu/127.14.147.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.147.212:44407
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790722213368738-14927-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790722213368738-14927-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790722213368738-14927-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 22:50:48.749194 23307 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 22:50:48.749425 23307 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 22:50:48.749451 23307 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 22:50:48.750747 23307 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 22:50:48.750795 23307 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 22:50:48.750824 23307 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.147.194
I20260929 22:50:48.752398 23307 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.147.212:44407
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790722213368738-14927-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790722213368738-14927-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790722213368738-14927-0/krb5kdc/kudu_127.14.147.194.keytab
--principal=kudu/127.14.147.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.147.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790722213368738-14927-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.14.147.194
--webserver_port=0
--tserver_master_addrs=127.14.147.254:40759
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23307
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.147.194
--log_dir=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790722213368738-14927-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 21:58:54 UTC on ccycloud.quasar-xguoti.root.comops.site
build id 4681390
I20260929 22:50:48.752642 23307 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 22:50:48.752863 23307 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 22:50:48.757716 23315 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 22:50:48.757790 23307 server_base.cc:1034] running on GCE node
W20260929 22:50:48.757709 23312 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 22:50:48.757709 23313 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 22:50:48.758060 23307 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 22:50:48.758282 23307 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 22:50:48.759449 23307 hybrid_clock.cc:648] HybridClock initialized: now 1790722248759433 us; error 30 us; skew 500 ppm
Sep 29 22:50:48 dist-test-slave-98mk krb5kdc[23064](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790722248, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.147.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 22:50:48.761844 23307 init.cc:377] Logged in from keytab as kudu/127.14.147.194@KRBTEST.COM (short username kudu)
I20260929 22:50:48.762296 23307 webserver.cc:460] Webserver started at http://127.14.147.194:46279/ using document root <none> and password file <none>
I20260929 22:50:48.762523 23307 fs_manager.cc:359] Metadata directory not provided
I20260929 22:50:48.762573 23307 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 22:50:48.762701 23307 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 22:50:48.763814 23307 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790722213368738-14927-0/minicluster-data/ts-1/data/instance:
uuid: "bd81ef5d69344ec0848ee65c71c52950"
format_stamp: "Formatted at 2026-09-29 22:50:48 on dist-test-slave-98mk"
server_key: "19b490c54ece7fdd6876c446c0bd311b"
server_key_iv: "914e78327b6723ba7c14b54fce07187d"
server_key_version: "encryptionkey@0"
I20260929 22:50:48.764129 23307 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790722213368738-14927-0/minicluster-data/ts-1/wal/instance:
uuid: "bd81ef5d69344ec0848ee65c71c52950"
format_stamp: "Formatted at 2026-09-29 22:50:48 on dist-test-slave-98mk"
server_key: "19b490c54ece7fdd6876c446c0bd311b"
server_key_iv: "914e78327b6723ba7c14b54fce07187d"
server_key_version: "encryptionkey@0"
I20260929 22:50:48.765403 23307 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260929 22:50:48.766043 23322 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 22:50:48.766201 23307 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260929 22:50:48.766280 23307 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790722213368738-14927-0/minicluster-data/ts-1/data,/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790722213368738-14927-0/minicluster-data/ts-1/wal
uuid: "bd81ef5d69344ec0848ee65c71c52950"
format_stamp: "Formatted at 2026-09-29 22:50:48 on dist-test-slave-98mk"
server_key: "19b490c54ece7fdd6876c446c0bd311b"
server_key_iv: "914e78327b6723ba7c14b54fce07187d"
server_key_version: "encryptionkey@0"
I20260929 22:50:48.766340 23307 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790722213368738-14927-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790722213368738-14927-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790722213368738-14927-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 22:50:48.776957 23307 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 22:50:48.778927 23307 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 22:50:48.779070 23307 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 22:50:48.779292 23307 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 22:50:48.779644 23307 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 22:50:48.779678 23307 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 22:50:48.779707 23307 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 22:50:48.779731 23307 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 22:50:48.785264 23307 rpc_server.cc:307] RPC server started. Bound to: 127.14.147.194:37677
I20260929 22:50:48.785316 23435 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.147.194:37677 every 8 connection(s)
I20260929 22:50:48.785629 23307 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790722213368738-14927-0/minicluster-data/ts-1/data/info.pb
Sep 29 22:50:48 dist-test-slave-98mk krb5kdc[23064](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790722248, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.147.194@KRBTEST.COM for kudu/127.14.147.254@KRBTEST.COM
I20260929 22:50:48.793292 14927 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhWOCLk/build/release/bin/kudu as pid 23307
I20260929 22:50:48.793421 14927 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790722213368738-14927-0/minicluster-data/ts-1/wal/instance
I20260929 22:50:48.793624 14927 external_mini_cluster.cc:1428] Setting key 339ebaef64e455f7425cee6cea971b31
I20260929 22:50:48.796171 23167 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:50:48.786356 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.147.194:56177 (local address 127.14.147.254:40759)
0929 22:50:48.786418 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0929 22:50:48.786421 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 22:50:48.786619 (+ 198us) server_negotiation.cc:407] Connection header received
0929 22:50:48.787125 (+ 506us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 22:50:48.787127 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 22:50:48.787138 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 22:50:48.787198 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0929 22:50:48.788468 (+ 1270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:50:48.788977 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:50:48.790121 (+ 1144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:50:48.790379 (+ 258us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:50:48.790467 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 22:50:48.793173 (+ 2706us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 22:50:48.793176 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 22:50:48.793176 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 22:50:48.793186 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0929 22:50:48.794686 (+ 1500us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 22:50:48.795089 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 22:50:48.795090 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 22:50:48.795091 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 22:50:48.795145 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 22:50:48.795505 (+ 360us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 22:50:48.795506 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 22:50:48.795506 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 22:50:48.795656 (+ 150us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 22:50:48.795725 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0929 22:50:48.795992 (+ 267us) server_negotiation.cc:299] Negotiation successful
0929 22:50:48.796077 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260929 22:50:48.796264 23438 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:50:48.786335 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.147.254:40759 (local address 127.14.147.194:56177)
0929 22:50:48.786507 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0929 22:50:48.786544 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0929 22:50:48.787013 (+ 469us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 22:50:48.787261 (+ 248us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 22:50:48.787265 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 22:50:48.787877 (+ 612us) client_negotiation.cc:189] Negotiated authn=SASL
0929 22:50:48.788307 (+ 430us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 22:50:48.788319 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 22:50:48.789114 (+ 795us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 22:50:48.789115 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 22:50:48.789975 (+ 860us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 22:50:48.789978 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 22:50:48.790488 (+ 510us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 22:50:48.790489 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 22:50:48.790616 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 22:50:48.791243 (+ 627us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 22:50:48.791254 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0929 22:50:48.793028 (+ 1774us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 22:50:48.794779 (+ 1751us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 22:50:48.794787 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 22:50:48.794790 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 22:50:48.795022 (+ 232us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 22:50:48.795230 (+ 208us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 22:50:48.795231 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 22:50:48.795232 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 22:50:48.795388 (+ 156us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 22:50:48.795719 (+ 331us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 22:50:48.795725 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 22:50:48.795859 (+ 134us) client_negotiation.cc:769] Sending connection context
0929 22:50:48.795946 (+ 87us) client_negotiation.cc:240] Negotiation successful
0929 22:50:48.796084 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":92,"threads_started":1}
I20260929 22:50:48.796748 23436 heartbeater.cc:347] Connected to a master server at 127.14.147.254:40759
I20260929 22:50:48.796852 23436 heartbeater.cc:464] Registering TS with master...
I20260929 22:50:48.797046 23436 heartbeater.cc:511] Master 127.14.147.254:40759 requested a full tablet report, sending...
I20260929 22:50:48.797410 23109 ts_manager.cc:194] Registered new tserver with Master: bd81ef5d69344ec0848ee65c71c52950 (127.14.147.194:37677)
I20260929 22:50:48.797780 23109 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.147.194@KRBTEST.COM'} at 127.14.147.194:56177
WARNING: no policy specified for kudu/127.14.147.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.147.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.147.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790722213368738-14927-0/krb5kdc/kudu_127.14.147.195.keytab.
Entry for principal kudu/127.14.147.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790722213368738-14927-0/krb5kdc/kudu_127.14.147.195.keytab.
I20260929 22:50:48.818219 14927 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhWOCLk/build/release/bin/kudu
/tmp/dist-test-taskhWOCLk/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790722213368738-14927-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790722213368738-14927-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790722213368738-14927-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790722213368738-14927-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.147.195:0
--local_ip_for_outbound_sockets=127.14.147.195
--webserver_interface=127.14.147.195
--webserver_port=0
--tserver_master_addrs=127.14.147.254:40759
--keytab_file=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790722213368738-14927-0/krb5kdc/kudu_127.14.147.195.keytab
--principal=kudu/127.14.147.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.147.212:44407
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790722213368738-14927-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790722213368738-14927-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790722213368738-14927-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 22:50:48.898891 23442 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 22:50:48.899106 23442 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 22:50:48.899130 23442 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 22:50:48.900346 23442 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 22:50:48.900393 23442 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 22:50:48.900418 23442 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.147.195
I20260929 22:50:48.901839 23442 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.147.212:44407
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790722213368738-14927-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790722213368738-14927-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790722213368738-14927-0/krb5kdc/kudu_127.14.147.195.keytab
--principal=kudu/127.14.147.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.147.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790722213368738-14927-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.14.147.195
--webserver_port=0
--tserver_master_addrs=127.14.147.254:40759
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23442
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.147.195
--log_dir=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790722213368738-14927-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 21:58:54 UTC on ccycloud.quasar-xguoti.root.comops.site
build id 4681390
I20260929 22:50:48.902038 23442 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 22:50:48.902231 23442 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260929 22:50:48.907196 23442 server_base.cc:1034] running on GCE node
W20260929 22:50:48.907133 23448 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 22:50:48.907198 23447 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 22:50:48.907133 23450 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 22:50:48.907490 23442 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 22:50:48.907753 23442 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 22:50:48.908914 23442 hybrid_clock.cc:648] HybridClock initialized: now 1790722248908902 us; error 24 us; skew 500 ppm
Sep 29 22:50:48 dist-test-slave-98mk krb5kdc[23064](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790722248, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.147.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 22:50:48.911345 23442 init.cc:377] Logged in from keytab as kudu/127.14.147.195@KRBTEST.COM (short username kudu)
I20260929 22:50:48.911818 23442 webserver.cc:460] Webserver started at http://127.14.147.195:32879/ using document root <none> and password file <none>
I20260929 22:50:48.912053 23442 fs_manager.cc:359] Metadata directory not provided
I20260929 22:50:48.912133 23442 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 22:50:48.912283 23442 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 22:50:48.913408 23442 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790722213368738-14927-0/minicluster-data/ts-2/data/instance:
uuid: "2325a0b0cf934dee80d69a49560ed32c"
format_stamp: "Formatted at 2026-09-29 22:50:48 on dist-test-slave-98mk"
server_key: "076c1c50cbf374f801515766cf133860"
server_key_iv: "480d117561144aa328cdf81aa95b4e10"
server_key_version: "encryptionkey@0"
I20260929 22:50:48.913738 23442 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790722213368738-14927-0/minicluster-data/ts-2/wal/instance:
uuid: "2325a0b0cf934dee80d69a49560ed32c"
format_stamp: "Formatted at 2026-09-29 22:50:48 on dist-test-slave-98mk"
server_key: "076c1c50cbf374f801515766cf133860"
server_key_iv: "480d117561144aa328cdf81aa95b4e10"
server_key_version: "encryptionkey@0"
I20260929 22:50:48.915073 23442 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260929 22:50:48.915937 23457 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 22:50:48.916110 23442 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260929 22:50:48.916205 23442 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790722213368738-14927-0/minicluster-data/ts-2/data,/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790722213368738-14927-0/minicluster-data/ts-2/wal
uuid: "2325a0b0cf934dee80d69a49560ed32c"
format_stamp: "Formatted at 2026-09-29 22:50:48 on dist-test-slave-98mk"
server_key: "076c1c50cbf374f801515766cf133860"
server_key_iv: "480d117561144aa328cdf81aa95b4e10"
server_key_version: "encryptionkey@0"
I20260929 22:50:48.916276 23442 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790722213368738-14927-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790722213368738-14927-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790722213368738-14927-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 22:50:48.928025 23442 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 22:50:48.930040 23442 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 22:50:48.930204 23442 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 22:50:48.930433 23442 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 22:50:48.930737 23442 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 22:50:48.930771 23442 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 22:50:48.930801 23442 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 22:50:48.930824 23442 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 22:50:48.936772 23442 rpc_server.cc:307] RPC server started. Bound to: 127.14.147.195:35505
I20260929 22:50:48.936812 23570 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.147.195:35505 every 8 connection(s)
I20260929 22:50:48.937131 23442 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790722213368738-14927-0/minicluster-data/ts-2/data/info.pb
Sep 29 22:50:48 dist-test-slave-98mk krb5kdc[23064](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790722248, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.147.195@KRBTEST.COM for kudu/127.14.147.254@KRBTEST.COM
I20260929 22:50:48.943022 14927 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhWOCLk/build/release/bin/kudu as pid 23442
I20260929 22:50:48.943126 14927 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790722213368738-14927-0/minicluster-data/ts-2/wal/instance
I20260929 22:50:48.943305 14927 external_mini_cluster.cc:1428] Setting key 2d46367ae1d95ed22b7b7d4ce539124a
I20260929 22:50:48.945757 23167 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:50:48.937780 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.147.195:34475 (local address 127.14.147.254:40759)
0929 22:50:48.937864 (+ 84us) server_negotiation.cc:206] Beginning negotiation
0929 22:50:48.937866 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 22:50:48.938026 (+ 160us) server_negotiation.cc:407] Connection header received
0929 22:50:48.938536 (+ 510us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 22:50:48.938537 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 22:50:48.938547 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 22:50:48.938607 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0929 22:50:48.939548 (+ 941us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:50:48.940053 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:50:48.941069 (+ 1016us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:50:48.941264 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:50:48.941330 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 22:50:48.943434 (+ 2104us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 22:50:48.943436 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 22:50:48.943436 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 22:50:48.943444 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0929 22:50:48.944701 (+ 1257us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 22:50:48.945048 (+ 347us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 22:50:48.945049 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 22:50:48.945049 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 22:50:48.945094 (+ 45us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 22:50:48.945270 (+ 176us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 22:50:48.945270 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 22:50:48.945271 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 22:50:48.945365 (+ 94us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 22:50:48.945411 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0929 22:50:48.945624 (+ 213us) server_negotiation.cc:299] Negotiation successful
0929 22:50:48.945683 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260929 22:50:48.945919 23574 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:50:48.937762 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.147.254:40759 (local address 127.14.147.195:34475)
0929 22:50:48.937912 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0929 22:50:48.937948 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0929 22:50:48.938429 (+ 481us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 22:50:48.938666 (+ 237us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 22:50:48.938669 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 22:50:48.939227 (+ 558us) client_negotiation.cc:189] Negotiated authn=SASL
0929 22:50:48.939466 (+ 239us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 22:50:48.939472 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 22:50:48.940156 (+ 684us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 22:50:48.940157 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 22:50:48.940980 (+ 823us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 22:50:48.940982 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 22:50:48.941333 (+ 351us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 22:50:48.941333 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 22:50:48.941442 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 22:50:48.941954 (+ 512us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 22:50:48.941962 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0929 22:50:48.943326 (+ 1364us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 22:50:48.944765 (+ 1439us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 22:50:48.944771 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 22:50:48.944773 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 22:50:48.944993 (+ 220us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 22:50:48.945138 (+ 145us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 22:50:48.945138 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 22:50:48.945138 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 22:50:48.945229 (+ 91us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 22:50:48.945407 (+ 178us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 22:50:48.945410 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 22:50:48.945525 (+ 115us) client_negotiation.cc:769] Sending connection context
0929 22:50:48.945603 (+ 78us) client_negotiation.cc:240] Negotiation successful
0929 22:50:48.945725 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":77,"threads_started":1}
I20260929 22:50:48.946307 23571 heartbeater.cc:347] Connected to a master server at 127.14.147.254:40759
I20260929 22:50:48.946399 23571 heartbeater.cc:464] Registering TS with master...
I20260929 22:50:48.946544 23571 heartbeater.cc:511] Master 127.14.147.254:40759 requested a full tablet report, sending...
I20260929 22:50:48.946846 23109 ts_manager.cc:194] Registered new tserver with Master: 2325a0b0cf934dee80d69a49560ed32c (127.14.147.195:35505)
I20260929 22:50:48.947207 23109 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.147.195@KRBTEST.COM'} at 127.14.147.195:34475
I20260929 22:50:48.955031 14927 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.14.147.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.14.147.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.14.147.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790722213368738-14927-0/krb5kdc/oryx_127.14.147.193.keytab.
Entry for principal oryx/127.14.147.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790722213368738-14927-0/krb5kdc/oryx_127.14.147.193.keytab.
WARNING: no policy specified for oryx/127.14.147.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.14.147.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.14.147.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790722213368738-14927-0/krb5kdc/oryx_127.14.147.194.keytab.
Entry for principal oryx/127.14.147.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790722213368738-14927-0/krb5kdc/oryx_127.14.147.194.keytab.
WARNING: no policy specified for oryx/127.14.147.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.14.147.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.14.147.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790722213368738-14927-0/krb5kdc/oryx_127.14.147.195.keytab.
Entry for principal oryx/127.14.147.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790722213368738-14927-0/krb5kdc/oryx_127.14.147.195.keytab.
I20260929 22:50:49.022233 14927 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhWOCLk/build/release/bin/kudu with pid 23171
I20260929 22:50:49.026428 14927 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhWOCLk/build/release/bin/kudu with pid 23307
I20260929 22:50:49.030584 14927 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhWOCLk/build/release/bin/kudu with pid 23442
I20260929 22:50:49.034669 14927 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhWOCLk/build/release/bin/kudu with pid 23077
I20260929 22:50:49.039340 14927 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhWOCLk/build/release/bin/kudu
/tmp/dist-test-taskhWOCLk/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790722213368738-14927-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790722213368738-14927-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790722213368738-14927-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790722213368738-14927-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.147.254:40759
--webserver_interface=127.14.147.254
--webserver_port=34221
--keytab_file=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790722213368738-14927-0/krb5kdc/kudu_127.14.147.254.keytab
--principal=kudu/127.14.147.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.147.212:44407
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.147.254:40759
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790722213368738-14927-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790722213368738-14927-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790722213368738-14927-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 22:50:49.120194 23581 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 22:50:49.120440 23581 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 22:50:49.120473 23581 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 22:50:49.121733 23581 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 22:50:49.121783 23581 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 22:50:49.121801 23581 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 22:50:49.121819 23581 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 22:50:49.121834 23581 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 22:50:49.123441 23581 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.147.212:44407
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790722213368738-14927-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790722213368738-14927-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.14.147.254:40759
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790722213368738-14927-0/krb5kdc/kudu_127.14.147.254.keytab
--principal=kudu/127.14.147.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.147.254:40759
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790722213368738-14927-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.14.147.254
--webserver_port=34221
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23581
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790722213368738-14927-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 21:58:54 UTC on ccycloud.quasar-xguoti.root.comops.site
build id 4681390
I20260929 22:50:49.123708 23581 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 22:50:49.123924 23581 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 22:50:49.128507 23589 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 22:50:49.128520 23586 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 22:50:49.128507 23587 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 22:50:49.128846 23581 server_base.cc:1034] running on GCE node
I20260929 22:50:49.129087 23581 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 22:50:49.129400 23581 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 22:50:49.130560 23581 hybrid_clock.cc:648] HybridClock initialized: now 1790722249130539 us; error 34 us; skew 500 ppm
Sep 29 22:50:49 dist-test-slave-98mk krb5kdc[23064](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790722249, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.147.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 22:50:49.133055 23581 init.cc:377] Logged in from keytab as kudu/127.14.147.254@KRBTEST.COM (short username kudu)
I20260929 22:50:49.133512 23581 webserver.cc:460] Webserver started at http://127.14.147.254:34221/ using document root <none> and password file <none>
I20260929 22:50:49.133731 23581 fs_manager.cc:359] Metadata directory not provided
I20260929 22:50:49.133793 23581 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 22:50:49.135113 23581 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260929 22:50:49.135800 23596 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 22:50:49.135950 23581 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260929 22:50:49.136036 23581 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790722213368738-14927-0/minicluster-data/master-0/data,/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790722213368738-14927-0/minicluster-data/master-0/wal
uuid: "316eee16b8584fbdb2deae855af51c1e"
format_stamp: "Formatted at 2026-09-29 22:50:48 on dist-test-slave-98mk"
server_key: "0de1e1db88ce4c47b4a95a07f051bdb8"
server_key_iv: "630657ccbddaaa82d241a91cfd869948"
server_key_version: "encryptionkey@0"
I20260929 22:50:49.136399 23581 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790722213368738-14927-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790722213368738-14927-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790722213368738-14927-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 22:50:49.182076 23581 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 22:50:49.184244 23581 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 22:50:49.184389 23581 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 22:50:49.188246 23581 rpc_server.cc:307] RPC server started. Bound to: 127.14.147.254:40759
I20260929 22:50:49.188292 23648 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.147.254:40759 every 8 connection(s)
I20260929 22:50:49.188807 23581 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790722213368738-14927-0/minicluster-data/master-0/data/info.pb
I20260929 22:50:49.190150 23649 sys_catalog.cc:263] Verifying existing consensus state
I20260929 22:50:49.190832 23649 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 316eee16b8584fbdb2deae855af51c1e: Bootstrap starting.
I20260929 22:50:49.194628 14927 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhWOCLk/build/release/bin/kudu as pid 23581
I20260929 22:50:49.195015 14927 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhWOCLk/build/release/bin/kudu
/tmp/dist-test-taskhWOCLk/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790722213368738-14927-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790722213368738-14927-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790722213368738-14927-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790722213368738-14927-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.147.193:36425
--local_ip_for_outbound_sockets=127.14.147.193
--tserver_master_addrs=127.14.147.254:40759
--webserver_port=35881
--webserver_interface=127.14.147.193
--keytab_file=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790722213368738-14927-0/krb5kdc/kudu_127.14.147.193.keytab
--principal=kudu/127.14.147.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.147.212:44407
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.14.147.193
--keytab_file=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790722213368738-14927-0/krb5kdc/oryx_127.14.147.193.keytab with env {KRB5CCNAME=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790722213368738-14927-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790722213368738-14927-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790722213368738-14927-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 22:50:49.199731 23649 log.cc:824] T 00000000000000000000000000000000 P 316eee16b8584fbdb2deae855af51c1e: Log is configured to *not* fsync() on all Append() calls
I20260929 22:50:49.201409 23649 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 316eee16b8584fbdb2deae855af51c1e: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260929 22:50:49.201689 23649 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 316eee16b8584fbdb2deae855af51c1e: Bootstrap complete.
I20260929 22:50:49.203876 23649 raft_consensus.cc:348] T 00000000000000000000000000000000 P 316eee16b8584fbdb2deae855af51c1e [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "316eee16b8584fbdb2deae855af51c1e" member_type: VOTER last_known_addr { host: "127.14.147.254" port: 40759 } }
I20260929 22:50:49.204113 23649 raft_consensus.cc:729] T 00000000000000000000000000000000 P 316eee16b8584fbdb2deae855af51c1e [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 316eee16b8584fbdb2deae855af51c1e, State: Initialized, Role: FOLLOWER
I20260929 22:50:49.204275 23649 consensus_queue.cc:260] T 00000000000000000000000000000000 P 316eee16b8584fbdb2deae855af51c1e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "316eee16b8584fbdb2deae855af51c1e" member_type: VOTER last_known_addr { host: "127.14.147.254" port: 40759 } }
I20260929 22:50:49.204342 23649 raft_consensus.cc:388] T 00000000000000000000000000000000 P 316eee16b8584fbdb2deae855af51c1e [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 22:50:49.204393 23649 raft_consensus.cc:482] T 00000000000000000000000000000000 P 316eee16b8584fbdb2deae855af51c1e [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 22:50:49.204438 23649 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 316eee16b8584fbdb2deae855af51c1e [term 1 FOLLOWER]: Advancing to term 2
I20260929 22:50:49.205482 23649 raft_consensus.cc:504] T 00000000000000000000000000000000 P 316eee16b8584fbdb2deae855af51c1e [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "316eee16b8584fbdb2deae855af51c1e" member_type: VOTER last_known_addr { host: "127.14.147.254" port: 40759 } }
I20260929 22:50:49.205576 23649 leader_election.cc:302] T 00000000000000000000000000000000 P 316eee16b8584fbdb2deae855af51c1e [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 316eee16b8584fbdb2deae855af51c1e; no voters:
I20260929 22:50:49.205746 23649 leader_election.cc:288] T 00000000000000000000000000000000 P 316eee16b8584fbdb2deae855af51c1e [CANDIDATE]: Term 2 election: Requested vote from peers
I20260929 22:50:49.205797 23653 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 316eee16b8584fbdb2deae855af51c1e [term 2 FOLLOWER]: Leader election won for term 2
I20260929 22:50:49.205916 23653 raft_consensus.cc:686] T 00000000000000000000000000000000 P 316eee16b8584fbdb2deae855af51c1e [term 2 LEADER]: Becoming Leader. State: Replica: 316eee16b8584fbdb2deae855af51c1e, State: Running, Role: LEADER
I20260929 22:50:49.206019 23649 sys_catalog.cc:564] T 00000000000000000000000000000000 P 316eee16b8584fbdb2deae855af51c1e [sys.catalog]: configured and running, proceeding with master startup.
I20260929 22:50:49.205997 23653 consensus_queue.cc:237] T 00000000000000000000000000000000 P 316eee16b8584fbdb2deae855af51c1e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "316eee16b8584fbdb2deae855af51c1e" member_type: VOTER last_known_addr { host: "127.14.147.254" port: 40759 } }
I20260929 22:50:49.206305 23653 sys_catalog.cc:455] T 00000000000000000000000000000000 P 316eee16b8584fbdb2deae855af51c1e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 316eee16b8584fbdb2deae855af51c1e. Latest consensus state: current_term: 2 leader_uuid: "316eee16b8584fbdb2deae855af51c1e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "316eee16b8584fbdb2deae855af51c1e" member_type: VOTER last_known_addr { host: "127.14.147.254" port: 40759 } } }
I20260929 22:50:49.206391 23653 sys_catalog.cc:458] T 00000000000000000000000000000000 P 316eee16b8584fbdb2deae855af51c1e [sys.catalog]: This master's current role is: LEADER
I20260929 22:50:49.206606 23655 sys_catalog.cc:455] T 00000000000000000000000000000000 P 316eee16b8584fbdb2deae855af51c1e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "316eee16b8584fbdb2deae855af51c1e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "316eee16b8584fbdb2deae855af51c1e" member_type: VOTER last_known_addr { host: "127.14.147.254" port: 40759 } } }
I20260929 22:50:49.206704 23655 sys_catalog.cc:458] T 00000000000000000000000000000000 P 316eee16b8584fbdb2deae855af51c1e [sys.catalog]: This master's current role is: LEADER
I20260929 22:50:49.206839 23658 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 22:50:49.207332 23658 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 22:50:49.207585 23658 catalog_manager.cc:1259] Loaded cluster ID: 80a35ce9412b4812b44420efd2b010e4
I20260929 22:50:49.207683 23658 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 22:50:49.208917 23658 catalog_manager.cc:1504] Loading token signing keys...
I20260929 22:50:49.209105 23658 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 316eee16b8584fbdb2deae855af51c1e: Loaded TSK: 0
I20260929 22:50:49.209323 23658 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260929 22:50:49.277850 23651 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 22:50:49.278074 23651 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 22:50:49.278098 23651 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 22:50:49.280128 23651 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 22:50:49.280189 23651 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 22:50:49.280222 23651 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.147.193
I20260929 22:50:49.282064 23651 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.147.212:44407
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790722213368738-14927-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790722213368738-14927-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790722213368738-14927-0/krb5kdc/oryx_127.14.147.193.keytab
--principal=oryx/127.14.147.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.147.193:36425
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790722213368738-14927-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.14.147.193
--webserver_port=35881
--tserver_master_addrs=127.14.147.254:40759
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23651
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.147.193
--log_dir=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790722213368738-14927-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 21:58:54 UTC on ccycloud.quasar-xguoti.root.comops.site
build id 4681390
I20260929 22:50:49.282296 23651 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 22:50:49.282534 23651 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 22:50:49.288298 23678 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 22:50:49.288308 23675 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 22:50:49.288395 23676 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 22:50:49.288738 23651 server_base.cc:1034] running on GCE node
I20260929 22:50:49.288935 23651 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 22:50:49.289165 23651 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 22:50:49.290341 23651 hybrid_clock.cc:648] HybridClock initialized: now 1790722249290324 us; error 32 us; skew 500 ppm
Sep 29 22:50:49 dist-test-slave-98mk krb5kdc[23064](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790722249, etypes {rep=17 tkt=17 ses=17}, oryx/127.14.147.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 22:50:49.293663 23651 init.cc:377] Logged in from keytab as oryx/127.14.147.193@KRBTEST.COM (short username oryx)
I20260929 22:50:49.294167 23651 webserver.cc:460] Webserver started at http://127.14.147.193:35881/ using document root <none> and password file <none>
I20260929 22:50:49.294390 23651 fs_manager.cc:359] Metadata directory not provided
I20260929 22:50:49.294456 23651 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 22:50:49.295868 23651 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260929 22:50:49.296438 23685 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 22:50:49.296662 23651 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260929 22:50:49.296785 23651 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790722213368738-14927-0/minicluster-data/ts-0/data,/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790722213368738-14927-0/minicluster-data/ts-0/wal
uuid: "5da67f43f97f46669eaf53392a39f370"
format_stamp: "Formatted at 2026-09-29 22:50:48 on dist-test-slave-98mk"
server_key: "d864017c8b75c5a0109a10d6b4979e83"
server_key_iv: "19909f39ec587f3ce111f2dc4f169eb5"
server_key_version: "encryptionkey@0"
I20260929 22:50:49.297269 23651 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790722213368738-14927-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790722213368738-14927-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790722213368738-14927-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 22:50:49.322206 23651 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 22:50:49.324707 23651 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 22:50:49.324867 23651 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 22:50:49.325139 23651 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 22:50:49.325489 23651 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 22:50:49.325529 23651 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 22:50:49.325564 23651 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 22:50:49.325585 23651 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 22:50:49.332762 23651 rpc_server.cc:307] RPC server started. Bound to: 127.14.147.193:36425
I20260929 22:50:49.332856 23799 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.147.193:36425 every 8 connection(s)
I20260929 22:50:49.333317 23651 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790722213368738-14927-0/minicluster-data/ts-0/data/info.pb
Sep 29 22:50:49 dist-test-slave-98mk krb5kdc[23064](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.147.193@KRBTEST.COM for oryx/127.14.147.254@KRBTEST.COM, Server not found in Kerberos database
I20260929 22:50:49.341048 14927 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhWOCLk/build/release/bin/kudu as pid 23651
Sep 29 22:50:49 dist-test-slave-98mk krb5kdc[23064](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.147.193@KRBTEST.COM for oryx/127.14.147.254@KRBTEST.COM, Server not found in Kerberos database
I20260929 22:50:49.341331 14927 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhWOCLk/build/release/bin/kudu
/tmp/dist-test-taskhWOCLk/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790722213368738-14927-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790722213368738-14927-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790722213368738-14927-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790722213368738-14927-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.147.194:37677
--local_ip_for_outbound_sockets=127.14.147.194
--tserver_master_addrs=127.14.147.254:40759
--webserver_port=46279
--webserver_interface=127.14.147.194
--keytab_file=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790722213368738-14927-0/krb5kdc/kudu_127.14.147.194.keytab
--principal=kudu/127.14.147.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.147.212:44407
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.14.147.194
--keytab_file=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790722213368738-14927-0/krb5kdc/oryx_127.14.147.194.keytab with env {KRB5CCNAME=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790722213368738-14927-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790722213368738-14927-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790722213368738-14927-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 22:50:49.342149 23802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:50:49.334205 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.147.254:40759 (local address 127.14.147.193:54959)
0929 22:50:49.334416 (+ 211us) negotiation.cc:107] Waiting for socket to connect
0929 22:50:49.334469 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0929 22:50:49.335002 (+ 533us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 22:50:49.335450 (+ 448us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 22:50:49.335454 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 22:50:49.336065 (+ 611us) client_negotiation.cc:189] Negotiated authn=SASL
0929 22:50:49.336279 (+ 214us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 22:50:49.336284 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 22:50:49.337363 (+ 1079us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 22:50:49.337364 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 22:50:49.338155 (+ 791us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 22:50:49.338157 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 22:50:49.338762 (+ 605us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 22:50:49.338763 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 22:50:49.338892 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 22:50:49.339436 (+ 544us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 22:50:49.339447 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0929 22:50:49.341952 (+ 2505us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.147.254:40759: Server oryx/127.14.147.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":148,"thread_start_us":96,"threads_started":1}
W20260929 22:50:49.342557 23803 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 22:50:49.334242 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.147.193:54959 (local address 127.14.147.254:40759)
0929 22:50:49.334462 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0929 22:50:49.334472 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 22:50:49.334579 (+ 107us) server_negotiation.cc:407] Connection header received
0929 22:50:49.335209 (+ 630us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 22:50:49.335215 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 22:50:49.335258 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 22:50:49.335372 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0929 22:50:49.336411 (+ 1039us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:50:49.337261 (+ 850us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:50:49.338238 (+ 977us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:50:49.338677 (+ 439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:50:49.338780 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 22:50:49.342117 (+ 3337us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.147.193:54959: BlockingRecv error: recv got EOF from 127.14.147.193:54959 (error 108)
Metrics: {"server-negotiator.queue_time_us":174,"thread_start_us":89,"threads_started":1}
W20260929 22:50:49.342734 23800 heartbeater.cc:644] Failed to heartbeat to 127.14.147.254:40759 (0 consecutive failures): Not authorized: Failed to ping master at 127.14.147.254:40759: Client connection negotiation failed: client connection to 127.14.147.254:40759: Server oryx/127.14.147.254@KRBTEST.COM not found in Kerberos database
Sep 29 22:50:49 dist-test-slave-98mk krb5kdc[23064](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.147.193@KRBTEST.COM for oryx/127.14.147.254@KRBTEST.COM, Server not found in Kerberos database
Sep 29 22:50:49 dist-test-slave-98mk krb5kdc[23064](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.147.193@KRBTEST.COM for oryx/127.14.147.254@KRBTEST.COM, Server not found in Kerberos database
I20260929 22:50:49.347476 23802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:50:49.342956 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.147.254:40759 (local address 127.14.147.193:57069)
0929 22:50:49.343029 (+ 73us) negotiation.cc:107] Waiting for socket to connect
0929 22:50:49.343040 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0929 22:50:49.343081 (+ 41us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 22:50:49.343274 (+ 193us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 22:50:49.343275 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 22:50:49.343446 (+ 171us) client_negotiation.cc:189] Negotiated authn=SASL
0929 22:50:49.343505 (+ 59us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 22:50:49.343507 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 22:50:49.344160 (+ 653us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 22:50:49.344161 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 22:50:49.344870 (+ 709us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 22:50:49.344873 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 22:50:49.345225 (+ 352us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 22:50:49.345225 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 22:50:49.345294 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 22:50:49.345758 (+ 464us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 22:50:49.345762 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0929 22:50:49.347372 (+ 1610us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.147.254:40759: Server oryx/127.14.147.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":56}
W20260929 22:50:49.347579 23803 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 22:50:49.343033 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.147.193:57069 (local address 127.14.147.254:40759)
0929 22:50:49.343173 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0929 22:50:49.343174 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0929 22:50:49.343180 (+ 6us) server_negotiation.cc:407] Connection header received
0929 22:50:49.343200 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 22:50:49.343201 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 22:50:49.343210 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 22:50:49.343258 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0929 22:50:49.343593 (+ 335us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:50:49.344088 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:50:49.344959 (+ 871us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:50:49.345154 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:50:49.345219 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 22:50:49.347504 (+ 2285us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.147.193:57069: BlockingRecv error: recv got EOF from 127.14.147.193:57069 (error 108)
Metrics: {"server-negotiator.queue_time_us":126,"thread_start_us":79,"threads_started":1}
Sep 29 22:50:49 dist-test-slave-98mk krb5kdc[23064](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.147.193@KRBTEST.COM for oryx/127.14.147.254@KRBTEST.COM, Server not found in Kerberos database
Sep 29 22:50:49 dist-test-slave-98mk krb5kdc[23064](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.147.193@KRBTEST.COM for oryx/127.14.147.254@KRBTEST.COM, Server not found in Kerberos database
I20260929 22:50:49.351953 23802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:50:49.347842 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.147.254:40759 (local address 127.14.147.193:34845)
0929 22:50:49.347886 (+ 44us) negotiation.cc:107] Waiting for socket to connect
0929 22:50:49.347897 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0929 22:50:49.347937 (+ 40us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 22:50:49.348048 (+ 111us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 22:50:49.348049 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 22:50:49.348171 (+ 122us) client_negotiation.cc:189] Negotiated authn=SASL
0929 22:50:49.348204 (+ 33us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 22:50:49.348205 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 22:50:49.348808 (+ 603us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 22:50:49.348809 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 22:50:49.349309 (+ 500us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 22:50:49.349310 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 22:50:49.349622 (+ 312us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 22:50:49.349622 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 22:50:49.349671 (+ 49us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 22:50:49.350140 (+ 469us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 22:50:49.350145 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0929 22:50:49.351893 (+ 1748us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.147.254:40759: Server oryx/127.14.147.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":27}
W20260929 22:50:49.352087 23800 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260929 22:50:49.352224 23803 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 22:50:49.347907 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.147.193:34845 (local address 127.14.147.254:40759)
0929 22:50:49.347951 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0929 22:50:49.347953 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 22:50:49.347961 (+ 8us) server_negotiation.cc:407] Connection header received
0929 22:50:49.347985 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 22:50:49.347986 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 22:50:49.347996 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 22:50:49.348040 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0929 22:50:49.348266 (+ 226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:50:49.348734 (+ 468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:50:49.349383 (+ 649us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:50:49.349564 (+ 181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:50:49.349619 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 22:50:49.352007 (+ 2388us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.147.193:34845: BlockingRecv error: recv got EOF from 127.14.147.193:34845 (error 108)
Metrics: {"server-negotiator.queue_time_us":21}
W20260929 22:50:49.424613 23804 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 22:50:49.424837 23804 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 22:50:49.424865 23804 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 22:50:49.426108 23804 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 22:50:49.426155 23804 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 22:50:49.426182 23804 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.147.194
I20260929 22:50:49.427798 23804 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.147.212:44407
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790722213368738-14927-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790722213368738-14927-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790722213368738-14927-0/krb5kdc/oryx_127.14.147.194.keytab
--principal=oryx/127.14.147.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.147.194:37677
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790722213368738-14927-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.14.147.194
--webserver_port=46279
--tserver_master_addrs=127.14.147.254:40759
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23804
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.147.194
--log_dir=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790722213368738-14927-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 21:58:54 UTC on ccycloud.quasar-xguoti.root.comops.site
build id 4681390
I20260929 22:50:49.428017 23804 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 22:50:49.428210 23804 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 22:50:49.433051 23811 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 22:50:49.433054 23810 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 22:50:49.433158 23804 server_base.cc:1034] running on GCE node
W20260929 22:50:49.433054 23813 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 22:50:49.433463 23804 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 22:50:49.433673 23804 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 22:50:49.434808 23804 hybrid_clock.cc:648] HybridClock initialized: now 1790722249434798 us; error 40 us; skew 500 ppm
Sep 29 22:50:49 dist-test-slave-98mk krb5kdc[23064](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790722249, etypes {rep=17 tkt=17 ses=17}, oryx/127.14.147.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 22:50:49.437250 23804 init.cc:377] Logged in from keytab as oryx/127.14.147.194@KRBTEST.COM (short username oryx)
I20260929 22:50:49.437686 23804 webserver.cc:460] Webserver started at http://127.14.147.194:46279/ using document root <none> and password file <none>
I20260929 22:50:49.437915 23804 fs_manager.cc:359] Metadata directory not provided
I20260929 22:50:49.438001 23804 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 22:50:49.439380 23804 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260929 22:50:49.439993 23820 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 22:50:49.440128 23804 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260929 22:50:49.440204 23804 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790722213368738-14927-0/minicluster-data/ts-1/data,/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790722213368738-14927-0/minicluster-data/ts-1/wal
uuid: "bd81ef5d69344ec0848ee65c71c52950"
format_stamp: "Formatted at 2026-09-29 22:50:48 on dist-test-slave-98mk"
server_key: "19b490c54ece7fdd6876c446c0bd311b"
server_key_iv: "914e78327b6723ba7c14b54fce07187d"
server_key_version: "encryptionkey@0"
I20260929 22:50:49.440565 23804 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790722213368738-14927-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790722213368738-14927-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790722213368738-14927-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 22:50:49.454339 23804 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 22:50:49.456593 23804 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 22:50:49.456717 23804 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 22:50:49.456971 23804 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 22:50:49.457262 23804 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 22:50:49.457296 23804 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 22:50:49.457319 23804 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 22:50:49.457341 23804 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 22:50:49.463003 23804 rpc_server.cc:307] RPC server started. Bound to: 127.14.147.194:37677
I20260929 22:50:49.463073 23934 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.147.194:37677 every 8 connection(s)
I20260929 22:50:49.463438 23804 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790722213368738-14927-0/minicluster-data/ts-1/data/info.pb
I20260929 22:50:49.467573 14927 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhWOCLk/build/release/bin/kudu as pid 23804
I20260929 22:50:49.467856 14927 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhWOCLk/build/release/bin/kudu
/tmp/dist-test-taskhWOCLk/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790722213368738-14927-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790722213368738-14927-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790722213368738-14927-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790722213368738-14927-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.147.195:35505
--local_ip_for_outbound_sockets=127.14.147.195
--tserver_master_addrs=127.14.147.254:40759
--webserver_port=32879
--webserver_interface=127.14.147.195
--keytab_file=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790722213368738-14927-0/krb5kdc/kudu_127.14.147.195.keytab
--principal=kudu/127.14.147.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.147.212:44407
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.14.147.195
--keytab_file=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790722213368738-14927-0/krb5kdc/oryx_127.14.147.195.keytab with env {KRB5CCNAME=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790722213368738-14927-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790722213368738-14927-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790722213368738-14927-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 29 22:50:49 dist-test-slave-98mk krb5kdc[23064](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.147.194@KRBTEST.COM for oryx/127.14.147.254@KRBTEST.COM, Server not found in Kerberos database
Sep 29 22:50:49 dist-test-slave-98mk krb5kdc[23064](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.147.194@KRBTEST.COM for oryx/127.14.147.254@KRBTEST.COM, Server not found in Kerberos database
W20260929 22:50:49.471115 23803 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 22:50:49.464208 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.147.194:46651 (local address 127.14.147.254:40759)
0929 22:50:49.464284 (+ 76us) server_negotiation.cc:206] Beginning negotiation
0929 22:50:49.464287 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 22:50:49.464485 (+ 198us) server_negotiation.cc:407] Connection header received
0929 22:50:49.464960 (+ 475us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 22:50:49.464961 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 22:50:49.464973 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 22:50:49.465030 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0929 22:50:49.466012 (+ 982us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:50:49.466505 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:50:49.467530 (+ 1025us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:50:49.467745 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:50:49.467805 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 22:50:49.471057 (+ 3252us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.147.194:46651: BlockingRecv error: recv got EOF from 127.14.147.194:46651 (error 108)
Metrics: {"server-negotiator.queue_time_us":45}
I20260929 22:50:49.471150 23937 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:50:49.464190 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.147.254:40759 (local address 127.14.147.194:46651)
0929 22:50:49.464359 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0929 22:50:49.464404 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0929 22:50:49.464886 (+ 482us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 22:50:49.465092 (+ 206us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 22:50:49.465095 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 22:50:49.465679 (+ 584us) client_negotiation.cc:189] Negotiated authn=SASL
0929 22:50:49.465891 (+ 212us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 22:50:49.465896 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 22:50:49.466585 (+ 689us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 22:50:49.466585 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 22:50:49.467456 (+ 871us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 22:50:49.467459 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 22:50:49.467817 (+ 358us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 22:50:49.467818 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 22:50:49.467984 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 22:50:49.468511 (+ 527us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 22:50:49.468521 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0929 22:50:49.470946 (+ 2425us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.147.254:40759: Server oryx/127.14.147.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":88,"threads_started":1}
W20260929 22:50:49.471375 23935 heartbeater.cc:644] Failed to heartbeat to 127.14.147.254:40759 (0 consecutive failures): Not authorized: Failed to ping master at 127.14.147.254:40759: Client connection negotiation failed: client connection to 127.14.147.254:40759: Server oryx/127.14.147.254@KRBTEST.COM not found in Kerberos database
Sep 29 22:50:49 dist-test-slave-98mk krb5kdc[23064](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.147.194@KRBTEST.COM for oryx/127.14.147.254@KRBTEST.COM, Server not found in Kerberos database
Sep 29 22:50:49 dist-test-slave-98mk krb5kdc[23064](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.147.194@KRBTEST.COM for oryx/127.14.147.254@KRBTEST.COM, Server not found in Kerberos database
I20260929 22:50:49.475731 23937 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:50:49.471606 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.147.254:40759 (local address 127.14.147.194:41477)
0929 22:50:49.471652 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0929 22:50:49.471662 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0929 22:50:49.471696 (+ 34us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 22:50:49.471863 (+ 167us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 22:50:49.471864 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 22:50:49.471982 (+ 118us) client_negotiation.cc:189] Negotiated authn=SASL
0929 22:50:49.472034 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 22:50:49.472035 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 22:50:49.472665 (+ 630us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 22:50:49.472666 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 22:50:49.473164 (+ 498us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 22:50:49.473166 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 22:50:49.473532 (+ 366us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 22:50:49.473533 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 22:50:49.473599 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 22:50:49.474060 (+ 461us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 22:50:49.474064 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0929 22:50:49.475675 (+ 1611us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.147.254:40759: Server oryx/127.14.147.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":27}
W20260929 22:50:49.475874 23803 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 22:50:49.471683 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.147.194:41477 (local address 127.14.147.254:40759)
0929 22:50:49.471757 (+ 74us) server_negotiation.cc:206] Beginning negotiation
0929 22:50:49.471759 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 22:50:49.471767 (+ 8us) server_negotiation.cc:407] Connection header received
0929 22:50:49.471793 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 22:50:49.471793 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 22:50:49.471803 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 22:50:49.471846 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0929 22:50:49.472099 (+ 253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:50:49.472589 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:50:49.473260 (+ 671us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:50:49.473473 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:50:49.473526 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 22:50:49.475807 (+ 2281us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.147.194:41477: BlockingRecv error: recv got EOF from 127.14.147.194:41477 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
Sep 29 22:50:49 dist-test-slave-98mk krb5kdc[23064](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.147.194@KRBTEST.COM for oryx/127.14.147.254@KRBTEST.COM, Server not found in Kerberos database
Sep 29 22:50:49 dist-test-slave-98mk krb5kdc[23064](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.147.194@KRBTEST.COM for oryx/127.14.147.254@KRBTEST.COM, Server not found in Kerberos database
I20260929 22:50:49.480083 23937 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:50:49.475974 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.147.254:40759 (local address 127.14.147.194:59711)
0929 22:50:49.476121 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0929 22:50:49.476130 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0929 22:50:49.476176 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 22:50:49.476289 (+ 113us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 22:50:49.476290 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 22:50:49.476413 (+ 123us) client_negotiation.cc:189] Negotiated authn=SASL
0929 22:50:49.476446 (+ 33us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 22:50:49.476448 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 22:50:49.477034 (+ 586us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 22:50:49.477035 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 22:50:49.477520 (+ 485us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 22:50:49.477521 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 22:50:49.477820 (+ 299us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 22:50:49.477821 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 22:50:49.477868 (+ 47us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 22:50:49.478297 (+ 429us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 22:50:49.478302 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0929 22:50:49.480008 (+ 1706us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.147.254:40759: Server oryx/127.14.147.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":135,"thread_start_us":99,"threads_started":1}
W20260929 22:50:49.480237 23803 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 22:50:49.476087 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.147.194:59711 (local address 127.14.147.254:40759)
0929 22:50:49.476124 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0929 22:50:49.476126 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 22:50:49.476183 (+ 57us) server_negotiation.cc:407] Connection header received
0929 22:50:49.476226 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 22:50:49.476226 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 22:50:49.476236 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 22:50:49.476281 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0929 22:50:49.476507 (+ 226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:50:49.476968 (+ 461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:50:49.477587 (+ 619us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:50:49.477764 (+ 177us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:50:49.477818 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 22:50:49.480105 (+ 2287us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.147.194:59711: BlockingRecv error: recv got EOF from 127.14.147.194:59711 (error 108)
Metrics: {"server-negotiator.queue_time_us":20}
W20260929 22:50:49.480223 23935 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260929 22:50:49.552067 23938 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 22:50:49.552305 23938 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 22:50:49.552340 23938 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 22:50:49.553617 23938 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 22:50:49.553670 23938 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 22:50:49.553709 23938 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.147.195
I20260929 22:50:49.555264 23938 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.147.212:44407
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790722213368738-14927-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790722213368738-14927-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790722213368738-14927-0/krb5kdc/oryx_127.14.147.195.keytab
--principal=oryx/127.14.147.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.147.195:35505
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790722213368738-14927-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.14.147.195
--webserver_port=32879
--tserver_master_addrs=127.14.147.254:40759
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23938
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.147.195
--log_dir=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790722213368738-14927-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 21:58:54 UTC on ccycloud.quasar-xguoti.root.comops.site
build id 4681390
I20260929 22:50:49.555568 23938 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 22:50:49.555811 23938 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 22:50:49.561000 23947 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 22:50:49.561000 23945 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 22:50:49.561017 23944 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 22:50:49.561256 23938 server_base.cc:1034] running on GCE node
I20260929 22:50:49.561484 23938 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 22:50:49.561719 23938 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 22:50:49.562889 23938 hybrid_clock.cc:648] HybridClock initialized: now 1790722249562869 us; error 33 us; skew 500 ppm
Sep 29 22:50:49 dist-test-slave-98mk krb5kdc[23064](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790722249, etypes {rep=17 tkt=17 ses=17}, oryx/127.14.147.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 22:50:49.565524 23938 init.cc:377] Logged in from keytab as oryx/127.14.147.195@KRBTEST.COM (short username oryx)
I20260929 22:50:49.566001 23938 webserver.cc:460] Webserver started at http://127.14.147.195:32879/ using document root <none> and password file <none>
I20260929 22:50:49.566210 23938 fs_manager.cc:359] Metadata directory not provided
I20260929 22:50:49.566269 23938 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 22:50:49.567660 23938 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260929 22:50:49.568182 23954 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 22:50:49.568317 23938 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260929 22:50:49.568396 23938 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790722213368738-14927-0/minicluster-data/ts-2/data,/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790722213368738-14927-0/minicluster-data/ts-2/wal
uuid: "2325a0b0cf934dee80d69a49560ed32c"
format_stamp: "Formatted at 2026-09-29 22:50:48 on dist-test-slave-98mk"
server_key: "076c1c50cbf374f801515766cf133860"
server_key_iv: "480d117561144aa328cdf81aa95b4e10"
server_key_version: "encryptionkey@0"
I20260929 22:50:49.568753 23938 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790722213368738-14927-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790722213368738-14927-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790722213368738-14927-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 22:50:49.590597 23938 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 22:50:49.592734 23938 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 22:50:49.592883 23938 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 22:50:49.593111 23938 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 22:50:49.593422 23938 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 22:50:49.593457 23938 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 22:50:49.593487 23938 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 22:50:49.593510 23938 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 22:50:49.598860 23938 rpc_server.cc:307] RPC server started. Bound to: 127.14.147.195:35505
I20260929 22:50:49.598906 24068 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.147.195:35505 every 8 connection(s)
I20260929 22:50:49.599237 23938 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790722213368738-14927-0/minicluster-data/ts-2/data/info.pb
I20260929 22:50:49.603626 14927 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhWOCLk/build/release/bin/kudu as pid 23938
Sep 29 22:50:49 dist-test-slave-98mk krb5kdc[23064](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.147.195@KRBTEST.COM for oryx/127.14.147.254@KRBTEST.COM, Server not found in Kerberos database
Sep 29 22:50:49 dist-test-slave-98mk krb5kdc[23064](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.147.195@KRBTEST.COM for oryx/127.14.147.254@KRBTEST.COM, Server not found in Kerberos database
W20260929 22:50:49.607048 23803 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 22:50:49.599971 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.147.195:58585 (local address 127.14.147.254:40759)
0929 22:50:49.600038 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0929 22:50:49.600040 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 22:50:49.600233 (+ 193us) server_negotiation.cc:407] Connection header received
0929 22:50:49.600741 (+ 508us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 22:50:49.600742 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 22:50:49.600754 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 22:50:49.600818 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0929 22:50:49.601810 (+ 992us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:50:49.602354 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:50:49.603300 (+ 946us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:50:49.603557 (+ 257us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:50:49.603618 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 22:50:49.606985 (+ 3367us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.147.195:58585: BlockingRecv error: recv got EOF from 127.14.147.195:58585 (error 108)
Metrics: {"server-negotiator.queue_time_us":38}
I20260929 22:50:49.607122 24071 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:50:49.599955 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.147.254:40759 (local address 127.14.147.195:58585)
0929 22:50:49.600119 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0929 22:50:49.600154 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0929 22:50:49.600644 (+ 490us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 22:50:49.600890 (+ 246us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 22:50:49.600894 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 22:50:49.601509 (+ 615us) client_negotiation.cc:189] Negotiated authn=SASL
0929 22:50:49.601704 (+ 195us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 22:50:49.601708 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 22:50:49.602437 (+ 729us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 22:50:49.602438 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 22:50:49.603215 (+ 777us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 22:50:49.603217 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 22:50:49.603636 (+ 419us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 22:50:49.603637 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 22:50:49.603773 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 22:50:49.604311 (+ 538us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 22:50:49.604321 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0929 22:50:49.606904 (+ 2583us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.147.254:40759: Server oryx/127.14.147.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":82,"threads_started":1}
W20260929 22:50:49.607362 24069 heartbeater.cc:644] Failed to heartbeat to 127.14.147.254:40759 (0 consecutive failures): Not authorized: Failed to ping master at 127.14.147.254:40759: Client connection negotiation failed: client connection to 127.14.147.254:40759: Server oryx/127.14.147.254@KRBTEST.COM not found in Kerberos database
Sep 29 22:50:49 dist-test-slave-98mk krb5kdc[23064](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.147.195@KRBTEST.COM for oryx/127.14.147.254@KRBTEST.COM, Server not found in Kerberos database
Sep 29 22:50:49 dist-test-slave-98mk krb5kdc[23064](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.147.195@KRBTEST.COM for oryx/127.14.147.254@KRBTEST.COM, Server not found in Kerberos database
I20260929 22:50:49.611536 24071 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:50:49.607585 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.147.254:40759 (local address 127.14.147.195:41471)
0929 22:50:49.607746 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0929 22:50:49.607756 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0929 22:50:49.607802 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 22:50:49.607913 (+ 111us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 22:50:49.607914 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 22:50:49.608053 (+ 139us) client_negotiation.cc:189] Negotiated authn=SASL
0929 22:50:49.608106 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 22:50:49.608109 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 22:50:49.608683 (+ 574us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 22:50:49.608684 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 22:50:49.609178 (+ 494us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 22:50:49.609179 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 22:50:49.609510 (+ 331us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 22:50:49.609511 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 22:50:49.609576 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 22:50:49.609880 (+ 304us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 22:50:49.609891 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0929 22:50:49.611461 (+ 1570us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.147.254:40759: Server oryx/127.14.147.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":151,"thread_start_us":98,"threads_started":1}
I20260929 22:50:49.611634 23805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:50:49.604350 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48052 (local address 127.14.147.254:40759)
0929 22:50:49.604451 (+ 101us) server_negotiation.cc:206] Beginning negotiation
0929 22:50:49.604453 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 22:50:49.604462 (+ 9us) server_negotiation.cc:407] Connection header received
0929 22:50:49.604482 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 22:50:49.604483 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 22:50:49.604495 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 22:50:49.604564 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0929 22:50:49.605264 (+ 700us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:50:49.605953 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:50:49.607086 (+ 1133us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:50:49.607360 (+ 274us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:50:49.607459 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 22:50:49.608672 (+ 1213us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 22:50:49.608676 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 22:50:49.608676 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 22:50:49.608688 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0929 22:50:49.610275 (+ 1587us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 22:50:49.610582 (+ 307us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 22:50:49.610585 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 22:50:49.610586 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 22:50:49.610707 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 22:50:49.610902 (+ 195us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 22:50:49.610903 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 22:50:49.610903 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 22:50:49.611137 (+ 234us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 22:50:49.611262 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0929 22:50:49.611382 (+ 120us) server_negotiation.cc:299] Negotiation successful
0929 22:50:49.611540 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
W20260929 22:50:49.611690 23803 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 22:50:49.607628 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.147.195:41471 (local address 127.14.147.254:40759)
0929 22:50:49.607665 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0929 22:50:49.607667 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 22:50:49.607800 (+ 133us) server_negotiation.cc:407] Connection header received
0929 22:50:49.607841 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 22:50:49.607842 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 22:50:49.607851 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 22:50:49.607903 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0929 22:50:49.608174 (+ 271us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:50:49.608611 (+ 437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:50:49.609249 (+ 638us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:50:49.609452 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:50:49.609503 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 22:50:49.611611 (+ 2108us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.147.195:41471: BlockingRecv error: recv got EOF from 127.14.147.195:41471 (error 108)
Metrics: {"mutex_wait_us":55,"server-negotiator.queue_time_us":19}
Sep 29 22:50:49 dist-test-slave-98mk krb5kdc[23064](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.147.195@KRBTEST.COM for oryx/127.14.147.254@KRBTEST.COM, Server not found in Kerberos database
Sep 29 22:50:49 dist-test-slave-98mk krb5kdc[23064](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.147.195@KRBTEST.COM for oryx/127.14.147.254@KRBTEST.COM, Server not found in Kerberos database
I20260929 22:50:49.616341 24071 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:50:49.612068 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.147.254:40759 (local address 127.14.147.195:37513)
0929 22:50:49.612147 (+ 79us) negotiation.cc:107] Waiting for socket to connect
0929 22:50:49.612157 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0929 22:50:49.612218 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 22:50:49.612344 (+ 126us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 22:50:49.612345 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 22:50:49.612472 (+ 127us) client_negotiation.cc:189] Negotiated authn=SASL
0929 22:50:49.612507 (+ 35us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 22:50:49.612509 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 22:50:49.613105 (+ 596us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 22:50:49.613105 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 22:50:49.613650 (+ 545us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 22:50:49.613652 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 22:50:49.613989 (+ 337us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 22:50:49.613990 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 22:50:49.614024 (+ 34us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 22:50:49.614405 (+ 381us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 22:50:49.614409 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0929 22:50:49.616286 (+ 1877us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.147.254:40759: Server oryx/127.14.147.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":65}
W20260929 22:50:49.616432 23803 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 22:50:49.612111 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.147.195:37513 (local address 127.14.147.254:40759)
0929 22:50:49.612150 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0929 22:50:49.612152 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 22:50:49.612193 (+ 41us) server_negotiation.cc:407] Connection header received
0929 22:50:49.612275 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 22:50:49.612276 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 22:50:49.612285 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 22:50:49.612339 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0929 22:50:49.612571 (+ 232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:50:49.613040 (+ 469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:50:49.613740 (+ 700us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:50:49.613930 (+ 190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:50:49.613987 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 22:50:49.616374 (+ 2387us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.147.195:37513: BlockingRecv error: recv got EOF from 127.14.147.195:37513 (error 108)
Metrics: {"server-negotiator.queue_time_us":17}
W20260929 22:50:49.616494 24069 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
Sep 29 22:50:50 dist-test-slave-98mk krb5kdc[23064](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.147.193@KRBTEST.COM for oryx/127.14.147.254@KRBTEST.COM, Server not found in Kerberos database
Sep 29 22:50:50 dist-test-slave-98mk krb5kdc[23064](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.147.193@KRBTEST.COM for oryx/127.14.147.254@KRBTEST.COM, Server not found in Kerberos database
I20260929 22:50:50.356966 24074 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:50:50.352512 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.147.254:40759 (local address 127.14.147.193:55643)
0929 22:50:50.352655 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0929 22:50:50.352666 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0929 22:50:50.352713 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 22:50:50.352902 (+ 189us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 22:50:50.352903 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 22:50:50.353070 (+ 167us) client_negotiation.cc:189] Negotiated authn=SASL
0929 22:50:50.353149 (+ 79us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 22:50:50.353151 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 22:50:50.353794 (+ 643us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 22:50:50.353795 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 22:50:50.354358 (+ 563us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 22:50:50.354360 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 22:50:50.354652 (+ 292us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 22:50:50.354653 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 22:50:50.354691 (+ 38us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 22:50:50.355112 (+ 421us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 22:50:50.355116 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0929 22:50:50.356891 (+ 1775us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.147.254:40759: Server oryx/127.14.147.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":62,"threads_started":1}
W20260929 22:50:50.357072 24075 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 22:50:50.352585 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.147.193:55643 (local address 127.14.147.254:40759)
0929 22:50:50.352710 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0929 22:50:50.352711 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0929 22:50:50.352721 (+ 10us) server_negotiation.cc:407] Connection header received
0929 22:50:50.352750 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 22:50:50.352750 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 22:50:50.352759 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 22:50:50.352830 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0929 22:50:50.353234 (+ 404us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:50:50.353731 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:50:50.354417 (+ 686us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:50:50.354604 (+ 187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:50:50.354648 (+ 44us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 22:50:50.357002 (+ 2354us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.147.193:55643: BlockingRecv error: recv got EOF from 127.14.147.193:55643 (error 108)
Metrics: {"server-negotiator.queue_time_us":106,"thread_start_us":68,"threads_started":1}
Sep 29 22:50:50 dist-test-slave-98mk krb5kdc[23064](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.147.194@KRBTEST.COM for oryx/127.14.147.254@KRBTEST.COM, Server not found in Kerberos database
Sep 29 22:50:50 dist-test-slave-98mk krb5kdc[23064](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.147.194@KRBTEST.COM for oryx/127.14.147.254@KRBTEST.COM, Server not found in Kerberos database
I20260929 22:50:50.485409 24076 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:50:50.480717 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.147.254:40759 (local address 127.14.147.194:44759)
0929 22:50:50.480871 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0929 22:50:50.480885 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 22:50:50.480953 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 22:50:50.481120 (+ 167us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 22:50:50.481122 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 22:50:50.481322 (+ 200us) client_negotiation.cc:189] Negotiated authn=SASL
0929 22:50:50.481377 (+ 55us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 22:50:50.481378 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 22:50:50.482033 (+ 655us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 22:50:50.482034 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 22:50:50.482606 (+ 572us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 22:50:50.482608 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 22:50:50.483026 (+ 418us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 22:50:50.483027 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 22:50:50.483086 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 22:50:50.483561 (+ 475us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 22:50:50.483565 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0929 22:50:50.485318 (+ 1753us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.147.254:40759: Server oryx/127.14.147.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":131,"thread_start_us":67,"threads_started":1}
W20260929 22:50:50.485487 24075 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 22:50:50.480827 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.147.194:44759 (local address 127.14.147.254:40759)
0929 22:50:50.480902 (+ 75us) server_negotiation.cc:206] Beginning negotiation
0929 22:50:50.480904 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 22:50:50.480938 (+ 34us) server_negotiation.cc:407] Connection header received
0929 22:50:50.481042 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 22:50:50.481043 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 22:50:50.481056 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 22:50:50.481104 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0929 22:50:50.481451 (+ 347us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:50:50.481953 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:50:50.482695 (+ 742us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:50:50.482952 (+ 257us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:50:50.483014 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 22:50:50.485431 (+ 2417us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.147.194:44759: BlockingRecv error: recv got EOF from 127.14.147.194:44759 (error 108)
Metrics: {"server-negotiator.queue_time_us":43}
Sep 29 22:50:50 dist-test-slave-98mk krb5kdc[23064](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.147.195@KRBTEST.COM for oryx/127.14.147.254@KRBTEST.COM, Server not found in Kerberos database
Sep 29 22:50:50 dist-test-slave-98mk krb5kdc[23064](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.147.195@KRBTEST.COM for oryx/127.14.147.254@KRBTEST.COM, Server not found in Kerberos database
I20260929 22:50:50.622015 24077 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:50:50.616971 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.147.254:40759 (local address 127.14.147.195:43077)
0929 22:50:50.617095 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0929 22:50:50.617108 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 22:50:50.617159 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 22:50:50.617296 (+ 137us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 22:50:50.617297 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 22:50:50.617460 (+ 163us) client_negotiation.cc:189] Negotiated authn=SASL
0929 22:50:50.617515 (+ 55us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 22:50:50.617517 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 22:50:50.618164 (+ 647us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 22:50:50.618166 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 22:50:50.618766 (+ 600us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 22:50:50.618768 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 22:50:50.619177 (+ 409us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 22:50:50.619179 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 22:50:50.619242 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 22:50:50.619680 (+ 438us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 22:50:50.619684 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0929 22:50:50.621912 (+ 2228us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.147.254:40759: Server oryx/127.14.147.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":103,"thread_start_us":71,"threads_started":1}
W20260929 22:50:50.622143 24075 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 22:50:50.617070 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.147.195:43077 (local address 127.14.147.254:40759)
0929 22:50:50.617137 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0929 22:50:50.617139 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 22:50:50.617151 (+ 12us) server_negotiation.cc:407] Connection header received
0929 22:50:50.617206 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 22:50:50.617207 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 22:50:50.617218 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 22:50:50.617278 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0929 22:50:50.617573 (+ 295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:50:50.618064 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:50:50.618862 (+ 798us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:50:50.619066 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:50:50.619128 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 22:50:50.622018 (+ 2890us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.147.195:43077: BlockingRecv error: recv got EOF from 127.14.147.195:43077 (error 108)
Metrics: {"server-negotiator.queue_time_us":42}
Sep 29 22:50:51 dist-test-slave-98mk krb5kdc[23064](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.147.193@KRBTEST.COM for oryx/127.14.147.254@KRBTEST.COM, Server not found in Kerberos database
Sep 29 22:50:51 dist-test-slave-98mk krb5kdc[23064](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.147.193@KRBTEST.COM for oryx/127.14.147.254@KRBTEST.COM, Server not found in Kerberos database
I20260929 22:50:51.362392 24078 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:50:51.357546 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.147.254:40759 (local address 127.14.147.193:41021)
0929 22:50:51.357705 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0929 22:50:51.357715 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0929 22:50:51.357766 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 22:50:51.357918 (+ 152us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 22:50:51.357918 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 22:50:51.358072 (+ 154us) client_negotiation.cc:189] Negotiated authn=SASL
0929 22:50:51.358121 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 22:50:51.358123 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 22:50:51.358909 (+ 786us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 22:50:51.358911 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 22:50:51.359666 (+ 755us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 22:50:51.359669 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 22:50:51.360034 (+ 365us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 22:50:51.360035 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 22:50:51.360072 (+ 37us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 22:50:51.360516 (+ 444us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 22:50:51.360521 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0929 22:50:51.362322 (+ 1801us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.147.254:40759: Server oryx/127.14.147.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":139,"thread_start_us":67,"threads_started":1}
W20260929 22:50:51.362567 24079 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 22:50:51.357634 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.147.193:41021 (local address 127.14.147.254:40759)
0929 22:50:51.357750 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0929 22:50:51.357751 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0929 22:50:51.357762 (+ 11us) server_negotiation.cc:407] Connection header received
0929 22:50:51.357838 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 22:50:51.357839 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 22:50:51.357851 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 22:50:51.357904 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0929 22:50:51.358220 (+ 316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:50:51.358826 (+ 606us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:50:51.359754 (+ 928us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:50:51.359968 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:50:51.360032 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 22:50:51.362493 (+ 2461us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.147.193:41021: BlockingRecv error: recv got EOF from 127.14.147.193:41021 (error 108)
Metrics: {"server-negotiator.queue_time_us":96,"thread_start_us":55,"threads_started":1}
Sep 29 22:50:51 dist-test-slave-98mk krb5kdc[23064](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.147.194@KRBTEST.COM for oryx/127.14.147.254@KRBTEST.COM, Server not found in Kerberos database
Sep 29 22:50:51 dist-test-slave-98mk krb5kdc[23064](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.147.194@KRBTEST.COM for oryx/127.14.147.254@KRBTEST.COM, Server not found in Kerberos database
I20260929 22:50:51.490628 24080 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:50:51.486093 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.147.254:40759 (local address 127.14.147.194:58019)
0929 22:50:51.486245 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0929 22:50:51.486254 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0929 22:50:51.486310 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 22:50:51.486459 (+ 149us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 22:50:51.486459 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 22:50:51.486615 (+ 156us) client_negotiation.cc:189] Negotiated authn=SASL
0929 22:50:51.486662 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 22:50:51.486664 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 22:50:51.487310 (+ 646us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 22:50:51.487311 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 22:50:51.487868 (+ 557us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 22:50:51.487871 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 22:50:51.488212 (+ 341us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 22:50:51.488213 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 22:50:51.488268 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 22:50:51.488712 (+ 444us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 22:50:51.488717 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0929 22:50:51.490566 (+ 1849us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.147.254:40759: Server oryx/127.14.147.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":134,"thread_start_us":84,"threads_started":1}
W20260929 22:50:51.490725 24079 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 22:50:51.486249 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.147.194:58019 (local address 127.14.147.254:40759)
0929 22:50:51.486317 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0929 22:50:51.486319 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 22:50:51.486330 (+ 11us) server_negotiation.cc:407] Connection header received
0929 22:50:51.486373 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 22:50:51.486374 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 22:50:51.486386 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 22:50:51.486439 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0929 22:50:51.486739 (+ 300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:50:51.487232 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:50:51.487956 (+ 724us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:50:51.488148 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:50:51.488203 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 22:50:51.490657 (+ 2454us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.147.194:58019: BlockingRecv error: recv got EOF from 127.14.147.194:58019 (error 108)
Metrics: {"server-negotiator.queue_time_us":42}
Sep 29 22:50:51 dist-test-slave-98mk krb5kdc[23064](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.147.195@KRBTEST.COM for oryx/127.14.147.254@KRBTEST.COM, Server not found in Kerberos database
Sep 29 22:50:51 dist-test-slave-98mk krb5kdc[23064](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.147.195@KRBTEST.COM for oryx/127.14.147.254@KRBTEST.COM, Server not found in Kerberos database
I20260929 22:50:51.627180 24081 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:50:51.622511 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.147.254:40759 (local address 127.14.147.195:42419)
0929 22:50:51.622702 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0929 22:50:51.622716 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 22:50:51.622779 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 22:50:51.622910 (+ 131us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 22:50:51.622910 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 22:50:51.623067 (+ 157us) client_negotiation.cc:189] Negotiated authn=SASL
0929 22:50:51.623118 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 22:50:51.623120 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 22:50:51.623871 (+ 751us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 22:50:51.623873 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 22:50:51.624454 (+ 581us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 22:50:51.624457 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 22:50:51.624861 (+ 404us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 22:50:51.624863 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 22:50:51.624915 (+ 52us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 22:50:51.625345 (+ 430us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 22:50:51.625350 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0929 22:50:51.627117 (+ 1767us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.147.254:40759: Server oryx/127.14.147.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":168,"thread_start_us":87,"threads_started":1}
W20260929 22:50:51.627290 24079 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 22:50:51.622685 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.147.195:42419 (local address 127.14.147.254:40759)
0929 22:50:51.622768 (+ 83us) server_negotiation.cc:206] Beginning negotiation
0929 22:50:51.622770 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 22:50:51.622780 (+ 10us) server_negotiation.cc:407] Connection header received
0929 22:50:51.622820 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 22:50:51.622821 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 22:50:51.622834 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 22:50:51.622891 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0929 22:50:51.623230 (+ 339us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:50:51.623786 (+ 556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:50:51.624572 (+ 786us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:50:51.624767 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:50:51.624823 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 22:50:51.627226 (+ 2403us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.147.195:42419: BlockingRecv error: recv got EOF from 127.14.147.195:42419 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
Sep 29 22:50:52 dist-test-slave-98mk krb5kdc[23064](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.147.193@KRBTEST.COM for oryx/127.14.147.254@KRBTEST.COM, Server not found in Kerberos database
Sep 29 22:50:52 dist-test-slave-98mk krb5kdc[23064](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.147.193@KRBTEST.COM for oryx/127.14.147.254@KRBTEST.COM, Server not found in Kerberos database
I20260929 22:50:52.367724 24082 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:50:52.362942 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.147.254:40759 (local address 127.14.147.193:49071)
0929 22:50:52.363100 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0929 22:50:52.363112 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0929 22:50:52.363171 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 22:50:52.363351 (+ 180us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 22:50:52.363353 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 22:50:52.363553 (+ 200us) client_negotiation.cc:189] Negotiated authn=SASL
0929 22:50:52.363602 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 22:50:52.363604 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 22:50:52.364319 (+ 715us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 22:50:52.364320 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 22:50:52.364839 (+ 519us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 22:50:52.364841 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 22:50:52.365240 (+ 399us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 22:50:52.365241 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 22:50:52.365293 (+ 52us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 22:50:52.365759 (+ 466us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 22:50:52.365764 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0929 22:50:52.367663 (+ 1899us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.147.254:40759: Server oryx/127.14.147.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":136,"thread_start_us":65,"threads_started":1}
W20260929 22:50:52.367851 24083 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 22:50:52.363058 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.147.193:49071 (local address 127.14.147.254:40759)
0929 22:50:52.363172 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0929 22:50:52.363174 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 22:50:52.363184 (+ 10us) server_negotiation.cc:407] Connection header received
0929 22:50:52.363225 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 22:50:52.363225 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 22:50:52.363236 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 22:50:52.363286 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0929 22:50:52.363694 (+ 408us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:50:52.364201 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:50:52.364942 (+ 741us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:50:52.365150 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:50:52.365211 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 22:50:52.367781 (+ 2570us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.147.193:49071: BlockingRecv error: recv got EOF from 127.14.147.193:49071 (error 108)
Metrics: {"server-negotiator.queue_time_us":92,"thread_start_us":48,"threads_started":1}
Sep 29 22:50:52 dist-test-slave-98mk krb5kdc[23064](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.147.194@KRBTEST.COM for oryx/127.14.147.254@KRBTEST.COM, Server not found in Kerberos database
Sep 29 22:50:52 dist-test-slave-98mk krb5kdc[23064](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.147.194@KRBTEST.COM for oryx/127.14.147.254@KRBTEST.COM, Server not found in Kerberos database
I20260929 22:50:52.496241 24084 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:50:52.491206 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.147.254:40759 (local address 127.14.147.194:54687)
0929 22:50:52.491348 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0929 22:50:52.491363 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 22:50:52.491439 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 22:50:52.491697 (+ 258us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 22:50:52.491698 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 22:50:52.491861 (+ 163us) client_negotiation.cc:189] Negotiated authn=SASL
0929 22:50:52.491910 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 22:50:52.491912 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 22:50:52.492633 (+ 721us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 22:50:52.492634 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 22:50:52.493274 (+ 640us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 22:50:52.493279 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 22:50:52.493769 (+ 490us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 22:50:52.493770 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 22:50:52.493819 (+ 49us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 22:50:52.494318 (+ 499us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 22:50:52.494323 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0929 22:50:52.496170 (+ 1847us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.147.254:40759: Server oryx/127.14.147.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":87,"threads_started":1}
W20260929 22:50:52.496448 24083 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 22:50:52.491369 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.147.194:54687 (local address 127.14.147.254:40759)
0929 22:50:52.491461 (+ 92us) server_negotiation.cc:206] Beginning negotiation
0929 22:50:52.491463 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 22:50:52.491475 (+ 12us) server_negotiation.cc:407] Connection header received
0929 22:50:52.491501 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 22:50:52.491502 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 22:50:52.491512 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 22:50:52.491585 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0929 22:50:52.492014 (+ 429us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:50:52.492535 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:50:52.493382 (+ 847us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:50:52.493593 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:50:52.493665 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 22:50:52.496398 (+ 2733us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.147.194:54687: BlockingRecv error: recv got EOF from 127.14.147.194:54687 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
Sep 29 22:50:52 dist-test-slave-98mk krb5kdc[23064](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.147.195@KRBTEST.COM for oryx/127.14.147.254@KRBTEST.COM, Server not found in Kerberos database
Sep 29 22:50:52 dist-test-slave-98mk krb5kdc[23064](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.147.195@KRBTEST.COM for oryx/127.14.147.254@KRBTEST.COM, Server not found in Kerberos database
I20260929 22:50:52.632588 24085 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:50:52.627763 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.147.254:40759 (local address 127.14.147.195:50641)
0929 22:50:52.627887 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0929 22:50:52.627898 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0929 22:50:52.627954 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 22:50:52.628096 (+ 142us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 22:50:52.628097 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 22:50:52.628269 (+ 172us) client_negotiation.cc:189] Negotiated authn=SASL
0929 22:50:52.628329 (+ 60us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 22:50:52.628331 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 22:50:52.628971 (+ 640us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 22:50:52.628972 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 22:50:52.629488 (+ 516us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 22:50:52.629490 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 22:50:52.629820 (+ 330us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 22:50:52.629822 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 22:50:52.629882 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 22:50:52.630383 (+ 501us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 22:50:52.630388 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0929 22:50:52.632527 (+ 2139us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.147.254:40759: Server oryx/127.14.147.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":103,"thread_start_us":57,"threads_started":1}
W20260929 22:50:52.632679 24083 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 22:50:52.627917 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.147.195:50641 (local address 127.14.147.254:40759)
0929 22:50:52.627984 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0929 22:50:52.627986 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 22:50:52.627997 (+ 11us) server_negotiation.cc:407] Connection header received
0929 22:50:52.628023 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 22:50:52.628024 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 22:50:52.628038 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 22:50:52.628083 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0929 22:50:52.628405 (+ 322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:50:52.628894 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:50:52.629569 (+ 675us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:50:52.629756 (+ 187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:50:52.629814 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 22:50:52.632619 (+ 2805us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.147.195:50641: BlockingRecv error: recv got EOF from 127.14.147.195:50641 (error 108)
Metrics: {"server-negotiator.queue_time_us":41}
Sep 29 22:50:53 dist-test-slave-98mk krb5kdc[23064](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.147.193@KRBTEST.COM for oryx/127.14.147.254@KRBTEST.COM, Server not found in Kerberos database
Sep 29 22:50:53 dist-test-slave-98mk krb5kdc[23064](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.147.193@KRBTEST.COM for oryx/127.14.147.254@KRBTEST.COM, Server not found in Kerberos database
I20260929 22:50:53.372542 24086 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:50:53.368227 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.147.254:40759 (local address 127.14.147.193:34837)
0929 22:50:53.368355 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0929 22:50:53.368368 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 22:50:53.368429 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 22:50:53.368552 (+ 123us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 22:50:53.368553 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 22:50:53.368703 (+ 150us) client_negotiation.cc:189] Negotiated authn=SASL
0929 22:50:53.368751 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 22:50:53.368753 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 22:50:53.369379 (+ 626us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 22:50:53.369379 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 22:50:53.369905 (+ 526us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 22:50:53.369907 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 22:50:53.370213 (+ 306us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 22:50:53.370214 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 22:50:53.370249 (+ 35us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 22:50:53.370658 (+ 409us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 22:50:53.370664 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0929 22:50:53.372462 (+ 1798us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.147.254:40759: Server oryx/127.14.147.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":106,"thread_start_us":67,"threads_started":1}
W20260929 22:50:53.372633 24087 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 22:50:53.368310 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.147.193:34837 (local address 127.14.147.254:40759)
0929 22:50:53.368415 (+ 105us) server_negotiation.cc:206] Beginning negotiation
0929 22:50:53.368417 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 22:50:53.368426 (+ 9us) server_negotiation.cc:407] Connection header received
0929 22:50:53.368485 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 22:50:53.368485 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 22:50:53.368495 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 22:50:53.368541 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0929 22:50:53.368805 (+ 264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:50:53.369307 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:50:53.369969 (+ 662us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:50:53.370155 (+ 186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:50:53.370210 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 22:50:53.372570 (+ 2360us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.147.193:34837: BlockingRecv error: recv got EOF from 127.14.147.193:34837 (error 108)
Metrics: {"server-negotiator.queue_time_us":84,"thread_start_us":53,"threads_started":1}
Sep 29 22:50:53 dist-test-slave-98mk krb5kdc[23064](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.147.194@KRBTEST.COM for oryx/127.14.147.254@KRBTEST.COM, Server not found in Kerberos database
Sep 29 22:50:53 dist-test-slave-98mk krb5kdc[23064](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.147.194@KRBTEST.COM for oryx/127.14.147.254@KRBTEST.COM, Server not found in Kerberos database
I20260929 22:50:53.501837 24088 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:50:53.496768 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.147.254:40759 (local address 127.14.147.194:36767)
0929 22:50:53.496920 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0929 22:50:53.496930 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0929 22:50:53.496989 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 22:50:53.497109 (+ 120us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 22:50:53.497109 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 22:50:53.497268 (+ 159us) client_negotiation.cc:189] Negotiated authn=SASL
0929 22:50:53.497321 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 22:50:53.497323 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 22:50:53.498103 (+ 780us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 22:50:53.498106 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 22:50:53.498740 (+ 634us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 22:50:53.498743 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 22:50:53.499337 (+ 594us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 22:50:53.499339 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 22:50:53.499483 (+ 144us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 22:50:53.499980 (+ 497us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 22:50:53.499984 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0929 22:50:53.501775 (+ 1791us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.147.254:40759: Server oryx/127.14.147.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":135,"thread_start_us":65,"threads_started":1}
W20260929 22:50:53.501979 24087 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 22:50:53.496862 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.147.194:36767 (local address 127.14.147.254:40759)
0929 22:50:53.496916 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0929 22:50:53.496918 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 22:50:53.496974 (+ 56us) server_negotiation.cc:407] Connection header received
0929 22:50:53.497034 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 22:50:53.497035 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 22:50:53.497046 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 22:50:53.497103 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0929 22:50:53.497413 (+ 310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:50:53.497936 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:50:53.498917 (+ 981us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:50:53.499186 (+ 269us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:50:53.499279 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 22:50:53.501914 (+ 2635us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.147.194:36767: BlockingRecv error: recv got EOF from 127.14.147.194:36767 (error 108)
Metrics: {"server-negotiator.queue_time_us":29}
Sep 29 22:50:53 dist-test-slave-98mk krb5kdc[23064](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.147.195@KRBTEST.COM for oryx/127.14.147.254@KRBTEST.COM, Server not found in Kerberos database
Sep 29 22:50:53 dist-test-slave-98mk krb5kdc[23064](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.147.195@KRBTEST.COM for oryx/127.14.147.254@KRBTEST.COM, Server not found in Kerberos database
I20260929 22:50:53.642872 24089 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:50:53.633151 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.147.254:40759 (local address 127.14.147.195:59777)
0929 22:50:53.633275 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0929 22:50:53.633286 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0929 22:50:53.633352 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 22:50:53.633472 (+ 120us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 22:50:53.633473 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 22:50:53.633630 (+ 157us) client_negotiation.cc:189] Negotiated authn=SASL
0929 22:50:53.633682 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 22:50:53.633683 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 22:50:53.634380 (+ 697us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 22:50:53.634381 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 22:50:53.635070 (+ 689us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 22:50:53.635074 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 22:50:53.635541 (+ 467us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 22:50:53.635544 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 22:50:53.635635 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 22:50:53.640975 (+ 5340us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 22:50:53.640982 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0929 22:50:53.642790 (+ 1808us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.147.254:40759: Server oryx/127.14.147.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":104,"thread_start_us":73,"threads_started":1}
W20260929 22:50:53.643010 24087 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 22:50:53.633278 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.147.195:59777 (local address 127.14.147.254:40759)
0929 22:50:53.633339 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0929 22:50:53.633341 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 22:50:53.633351 (+ 10us) server_negotiation.cc:407] Connection header received
0929 22:50:53.633403 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 22:50:53.633404 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 22:50:53.633416 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 22:50:53.633459 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0929 22:50:53.633736 (+ 277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:50:53.634315 (+ 579us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:50:53.635142 (+ 827us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:50:53.635336 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:50:53.635402 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 22:50:53.642943 (+ 7541us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.147.195:59777: BlockingRecv error: recv got EOF from 127.14.147.195:59777 (error 108)
Metrics: {"server-negotiator.queue_time_us":36}
Sep 29 22:50:54 dist-test-slave-98mk krb5kdc[23064](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.147.193@KRBTEST.COM for oryx/127.14.147.254@KRBTEST.COM, Server not found in Kerberos database
Sep 29 22:50:54 dist-test-slave-98mk krb5kdc[23064](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.147.193@KRBTEST.COM for oryx/127.14.147.254@KRBTEST.COM, Server not found in Kerberos database
I20260929 22:50:54.377223 24090 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:50:54.373016 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.147.254:40759 (local address 127.14.147.193:48839)
0929 22:50:54.373143 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0929 22:50:54.373156 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 22:50:54.373218 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 22:50:54.373394 (+ 176us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 22:50:54.373395 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 22:50:54.373550 (+ 155us) client_negotiation.cc:189] Negotiated authn=SASL
0929 22:50:54.373597 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 22:50:54.373598 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 22:50:54.374196 (+ 598us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 22:50:54.374197 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 22:50:54.374709 (+ 512us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 22:50:54.374711 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 22:50:54.375086 (+ 375us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 22:50:54.375087 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 22:50:54.375123 (+ 36us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 22:50:54.375567 (+ 444us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 22:50:54.375571 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0929 22:50:54.377163 (+ 1592us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.147.254:40759: Server oryx/127.14.147.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":106,"thread_start_us":55,"threads_started":1}
W20260929 22:50:54.377349 24091 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 22:50:54.373083 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.147.193:48839 (local address 127.14.147.254:40759)
0929 22:50:54.373271 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0929 22:50:54.373272 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0929 22:50:54.373282 (+ 10us) server_negotiation.cc:407] Connection header received
0929 22:50:54.373304 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 22:50:54.373305 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 22:50:54.373315 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 22:50:54.373386 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0929 22:50:54.373650 (+ 264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:50:54.374143 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:50:54.374822 (+ 679us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:50:54.375026 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:50:54.375087 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 22:50:54.377260 (+ 2173us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.147.193:48839: BlockingRecv error: recv got EOF from 127.14.147.193:48839 (error 108)
Metrics: {"server-negotiator.queue_time_us":166,"thread_start_us":66,"threads_started":1}
Sep 29 22:50:54 dist-test-slave-98mk krb5kdc[23064](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.147.194@KRBTEST.COM for oryx/127.14.147.254@KRBTEST.COM, Server not found in Kerberos database
Sep 29 22:50:54 dist-test-slave-98mk krb5kdc[23064](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.147.194@KRBTEST.COM for oryx/127.14.147.254@KRBTEST.COM, Server not found in Kerberos database
I20260929 22:50:54.506922 24092 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:50:54.502284 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.147.254:40759 (local address 127.14.147.194:36109)
0929 22:50:54.502456 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0929 22:50:54.502467 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0929 22:50:54.502514 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 22:50:54.502640 (+ 126us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 22:50:54.502641 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 22:50:54.502818 (+ 177us) client_negotiation.cc:189] Negotiated authn=SASL
0929 22:50:54.502869 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 22:50:54.502871 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 22:50:54.503500 (+ 629us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 22:50:54.503501 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 22:50:54.504054 (+ 553us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 22:50:54.504056 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 22:50:54.504437 (+ 381us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 22:50:54.504438 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 22:50:54.504492 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 22:50:54.504925 (+ 433us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 22:50:54.504929 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0929 22:50:54.506840 (+ 1911us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.147.254:40759: Server oryx/127.14.147.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":153,"thread_start_us":92,"threads_started":1}
W20260929 22:50:54.507001 24091 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 22:50:54.502430 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.147.194:36109 (local address 127.14.147.254:40759)
0929 22:50:54.502503 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0929 22:50:54.502505 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 22:50:54.502513 (+ 8us) server_negotiation.cc:407] Connection header received
0929 22:50:54.502564 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 22:50:54.502565 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 22:50:54.502576 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 22:50:54.502625 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0929 22:50:54.502934 (+ 309us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:50:54.503438 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:50:54.504146 (+ 708us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:50:54.504342 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:50:54.504400 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 22:50:54.506947 (+ 2547us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.147.194:36109: BlockingRecv error: recv got EOF from 127.14.147.194:36109 (error 108)
Metrics: {"server-negotiator.queue_time_us":44}
Sep 29 22:50:54 dist-test-slave-98mk krb5kdc[23064](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.147.195@KRBTEST.COM for oryx/127.14.147.254@KRBTEST.COM, Server not found in Kerberos database
Sep 29 22:50:54 dist-test-slave-98mk krb5kdc[23064](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.147.195@KRBTEST.COM for oryx/127.14.147.254@KRBTEST.COM, Server not found in Kerberos database
I20260929 22:50:54.647766 24093 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:50:54.643340 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.147.254:40759 (local address 127.14.147.195:58443)
0929 22:50:54.643518 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0929 22:50:54.643529 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0929 22:50:54.643595 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 22:50:54.643718 (+ 123us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 22:50:54.643719 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 22:50:54.643876 (+ 157us) client_negotiation.cc:189] Negotiated authn=SASL
0929 22:50:54.643927 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 22:50:54.643929 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 22:50:54.644698 (+ 769us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 22:50:54.644699 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 22:50:54.645219 (+ 520us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 22:50:54.645220 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 22:50:54.645509 (+ 289us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 22:50:54.645509 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 22:50:54.645546 (+ 37us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 22:50:54.645937 (+ 391us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 22:50:54.645942 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0929 22:50:54.647705 (+ 1763us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.147.254:40759: Server oryx/127.14.147.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":157,"thread_start_us":107,"threads_started":1}
W20260929 22:50:54.647904 24091 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 22:50:54.643410 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.147.195:58443 (local address 127.14.147.254:40759)
0929 22:50:54.643483 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0929 22:50:54.643485 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 22:50:54.643593 (+ 108us) server_negotiation.cc:407] Connection header received
0929 22:50:54.643645 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 22:50:54.643645 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 22:50:54.643657 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 22:50:54.643706 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0929 22:50:54.644017 (+ 311us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:50:54.644602 (+ 585us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:50:54.645286 (+ 684us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:50:54.645462 (+ 176us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:50:54.645505 (+ 43us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 22:50:54.647828 (+ 2323us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.147.195:58443: BlockingRecv error: recv got EOF from 127.14.147.195:58443 (error 108)
Metrics: {"server-negotiator.queue_time_us":23}
Sep 29 22:50:55 dist-test-slave-98mk krb5kdc[23064](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.147.193@KRBTEST.COM for oryx/127.14.147.254@KRBTEST.COM, Server not found in Kerberos database
Sep 29 22:50:55 dist-test-slave-98mk krb5kdc[23064](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.147.193@KRBTEST.COM for oryx/127.14.147.254@KRBTEST.COM, Server not found in Kerberos database
I20260929 22:50:55.382242 24094 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:50:55.377697 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.147.254:40759 (local address 127.14.147.193:35889)
0929 22:50:55.377823 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0929 22:50:55.377834 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0929 22:50:55.377894 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 22:50:55.378048 (+ 154us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 22:50:55.378049 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 22:50:55.378244 (+ 195us) client_negotiation.cc:189] Negotiated authn=SASL
0929 22:50:55.378304 (+ 60us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 22:50:55.378307 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 22:50:55.378994 (+ 687us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 22:50:55.378995 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 22:50:55.379557 (+ 562us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 22:50:55.379559 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 22:50:55.379907 (+ 348us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 22:50:55.379908 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 22:50:55.379944 (+ 36us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 22:50:55.380406 (+ 462us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 22:50:55.380410 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0929 22:50:55.382181 (+ 1771us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.147.254:40759: Server oryx/127.14.147.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":105,"thread_start_us":53,"threads_started":1}
W20260929 22:50:55.382398 24095 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 22:50:55.377791 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.147.193:35889 (local address 127.14.147.254:40759)
0929 22:50:55.377911 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0929 22:50:55.377913 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 22:50:55.377923 (+ 10us) server_negotiation.cc:407] Connection header received
0929 22:50:55.377963 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 22:50:55.377964 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 22:50:55.377976 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 22:50:55.378035 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0929 22:50:55.378404 (+ 369us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:50:55.378908 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:50:55.379643 (+ 735us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:50:55.379846 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:50:55.379900 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 22:50:55.382318 (+ 2418us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.147.193:35889: BlockingRecv error: recv got EOF from 127.14.147.193:35889 (error 108)
Metrics: {"server-negotiator.queue_time_us":95,"thread_start_us":49,"threads_started":1}
Sep 29 22:50:55 dist-test-slave-98mk krb5kdc[23064](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.147.194@KRBTEST.COM for oryx/127.14.147.254@KRBTEST.COM, Server not found in Kerberos database
Sep 29 22:50:55 dist-test-slave-98mk krb5kdc[23064](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.147.194@KRBTEST.COM for oryx/127.14.147.254@KRBTEST.COM, Server not found in Kerberos database
I20260929 22:50:55.513252 24096 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:50:55.507462 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.147.254:40759 (local address 127.14.147.194:59059)
0929 22:50:55.507599 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0929 22:50:55.507612 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 22:50:55.507714 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 22:50:55.507914 (+ 200us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 22:50:55.507915 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 22:50:55.508134 (+ 219us) client_negotiation.cc:189] Negotiated authn=SASL
0929 22:50:55.508196 (+ 62us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 22:50:55.508198 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 22:50:55.509138 (+ 940us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 22:50:55.509139 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 22:50:55.509947 (+ 808us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 22:50:55.509950 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 22:50:55.510479 (+ 529us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 22:50:55.510480 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 22:50:55.510544 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 22:50:55.511000 (+ 456us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 22:50:55.511005 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0929 22:50:55.513173 (+ 2168us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.147.254:40759: Server oryx/127.14.147.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":68,"threads_started":1}
W20260929 22:50:55.513430 24095 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 22:50:55.507574 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.147.194:59059 (local address 127.14.147.254:40759)
0929 22:50:55.507635 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0929 22:50:55.507637 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 22:50:55.507714 (+ 77us) server_negotiation.cc:407] Connection header received
0929 22:50:55.507818 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 22:50:55.507820 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 22:50:55.507834 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 22:50:55.507896 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0929 22:50:55.508269 (+ 373us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:50:55.509065 (+ 796us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:50:55.510010 (+ 945us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:50:55.510346 (+ 336us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:50:55.510418 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 22:50:55.513308 (+ 2890us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.147.194:59059: BlockingRecv error: recv got EOF from 127.14.147.194:59059 (error 108)
Metrics: {"server-negotiator.queue_time_us":35}
Sep 29 22:50:55 dist-test-slave-98mk krb5kdc[23064](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.147.195@KRBTEST.COM for oryx/127.14.147.254@KRBTEST.COM, Server not found in Kerberos database
Sep 29 22:50:55 dist-test-slave-98mk krb5kdc[23064](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.147.195@KRBTEST.COM for oryx/127.14.147.254@KRBTEST.COM, Server not found in Kerberos database
I20260929 22:50:55.652819 24097 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:50:55.648260 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.147.254:40759 (local address 127.14.147.195:37967)
0929 22:50:55.648410 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0929 22:50:55.648424 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 22:50:55.648511 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 22:50:55.648656 (+ 145us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 22:50:55.648657 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 22:50:55.648836 (+ 179us) client_negotiation.cc:189] Negotiated authn=SASL
0929 22:50:55.648882 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 22:50:55.648884 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 22:50:55.649521 (+ 637us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 22:50:55.649522 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 22:50:55.650034 (+ 512us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 22:50:55.650036 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 22:50:55.650364 (+ 328us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 22:50:55.650365 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 22:50:55.650415 (+ 50us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 22:50:55.650915 (+ 500us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 22:50:55.650921 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0929 22:50:55.652750 (+ 1829us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.147.254:40759: Server oryx/127.14.147.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":78,"threads_started":1}
W20260929 22:50:55.652896 24095 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 22:50:55.648358 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.147.195:37967 (local address 127.14.147.254:40759)
0929 22:50:55.648448 (+ 90us) server_negotiation.cc:206] Beginning negotiation
0929 22:50:55.648450 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 22:50:55.648483 (+ 33us) server_negotiation.cc:407] Connection header received
0929 22:50:55.648578 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 22:50:55.648579 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 22:50:55.648595 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 22:50:55.648641 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0929 22:50:55.648953 (+ 312us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:50:55.649450 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:50:55.650109 (+ 659us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:50:55.650301 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:50:55.650365 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 22:50:55.652839 (+ 2474us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.147.195:37967: BlockingRecv error: recv got EOF from 127.14.147.195:37967 (error 108)
Metrics: {"server-negotiator.queue_time_us":63}
Sep 29 22:50:56 dist-test-slave-98mk krb5kdc[23064](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.147.193@KRBTEST.COM for oryx/127.14.147.254@KRBTEST.COM, Server not found in Kerberos database
Sep 29 22:50:56 dist-test-slave-98mk krb5kdc[23064](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.147.193@KRBTEST.COM for oryx/127.14.147.254@KRBTEST.COM, Server not found in Kerberos database
I20260929 22:50:56.387336 24098 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:50:56.382782 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.147.254:40759 (local address 127.14.147.193:54221)
0929 22:50:56.382940 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0929 22:50:56.382951 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0929 22:50:56.383000 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 22:50:56.383136 (+ 136us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 22:50:56.383137 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 22:50:56.383298 (+ 161us) client_negotiation.cc:189] Negotiated authn=SASL
0929 22:50:56.383351 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 22:50:56.383353 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 22:50:56.383997 (+ 644us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 22:50:56.383997 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 22:50:56.384553 (+ 556us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 22:50:56.384555 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 22:50:56.384868 (+ 313us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 22:50:56.384869 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 22:50:56.384910 (+ 41us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 22:50:56.385338 (+ 428us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 22:50:56.385343 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0929 22:50:56.387253 (+ 1910us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.147.254:40759: Server oryx/127.14.147.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":139,"thread_start_us":68,"threads_started":1}
W20260929 22:50:56.387449 24099 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 22:50:56.382879 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.147.193:54221 (local address 127.14.147.254:40759)
0929 22:50:56.383009 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0929 22:50:56.383011 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 22:50:56.383019 (+ 8us) server_negotiation.cc:407] Connection header received
0929 22:50:56.383044 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 22:50:56.383045 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 22:50:56.383055 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 22:50:56.383122 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0929 22:50:56.383434 (+ 312us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:50:56.383931 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:50:56.384625 (+ 694us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:50:56.384812 (+ 187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:50:56.384860 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 22:50:56.387363 (+ 2503us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.147.193:54221: BlockingRecv error: recv got EOF from 127.14.147.193:54221 (error 108)
Metrics: {"server-negotiator.queue_time_us":110,"thread_start_us":69,"threads_started":1}
Sep 29 22:50:56 dist-test-slave-98mk krb5kdc[23064](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.147.194@KRBTEST.COM for oryx/127.14.147.254@KRBTEST.COM, Server not found in Kerberos database
Sep 29 22:50:56 dist-test-slave-98mk krb5kdc[23064](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.147.194@KRBTEST.COM for oryx/127.14.147.254@KRBTEST.COM, Server not found in Kerberos database
I20260929 22:50:56.518600 24100 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:50:56.513759 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.147.254:40759 (local address 127.14.147.194:42095)
0929 22:50:56.513887 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0929 22:50:56.513899 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0929 22:50:56.513963 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 22:50:56.514155 (+ 192us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 22:50:56.514156 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 22:50:56.514316 (+ 160us) client_negotiation.cc:189] Negotiated authn=SASL
0929 22:50:56.514370 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 22:50:56.514372 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 22:50:56.514987 (+ 615us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 22:50:56.514988 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 22:50:56.515571 (+ 583us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 22:50:56.515573 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 22:50:56.515916 (+ 343us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 22:50:56.515917 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 22:50:56.515972 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 22:50:56.516488 (+ 516us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 22:50:56.516493 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0929 22:50:56.518517 (+ 2024us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.147.254:40759: Server oryx/127.14.147.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":106,"thread_start_us":59,"threads_started":1}
W20260929 22:50:56.518967 24099 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 22:50:56.513844 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.147.194:42095 (local address 127.14.147.254:40759)
0929 22:50:56.513937 (+ 93us) server_negotiation.cc:206] Beginning negotiation
0929 22:50:56.513939 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 22:50:56.513950 (+ 11us) server_negotiation.cc:407] Connection header received
0929 22:50:56.514023 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 22:50:56.514024 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 22:50:56.514034 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 22:50:56.514081 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0929 22:50:56.514434 (+ 353us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:50:56.514932 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:50:56.515627 (+ 695us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:50:56.515812 (+ 185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:50:56.515865 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 22:50:56.518908 (+ 3043us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.147.194:42095: BlockingRecv error: recv got EOF from 127.14.147.194:42095 (error 108)
Metrics: {"server-negotiator.queue_time_us":65}
Sep 29 22:50:56 dist-test-slave-98mk krb5kdc[23064](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.147.195@KRBTEST.COM for oryx/127.14.147.254@KRBTEST.COM, Server not found in Kerberos database
Sep 29 22:50:56 dist-test-slave-98mk krb5kdc[23064](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.147.195@KRBTEST.COM for oryx/127.14.147.254@KRBTEST.COM, Server not found in Kerberos database
I20260929 22:50:56.657586 24101 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:50:56.653275 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.147.254:40759 (local address 127.14.147.195:45217)
0929 22:50:56.653433 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0929 22:50:56.653444 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0929 22:50:56.653499 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 22:50:56.653614 (+ 115us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 22:50:56.653615 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 22:50:56.653766 (+ 151us) client_negotiation.cc:189] Negotiated authn=SASL
0929 22:50:56.653819 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 22:50:56.653822 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 22:50:56.654506 (+ 684us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 22:50:56.654507 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 22:50:56.655049 (+ 542us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 22:50:56.655051 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 22:50:56.655361 (+ 310us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 22:50:56.655363 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 22:50:56.655442 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 22:50:56.655863 (+ 421us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 22:50:56.655868 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0929 22:50:56.657523 (+ 1655us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.147.254:40759: Server oryx/127.14.147.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":142,"thread_start_us":81,"threads_started":1}
W20260929 22:50:56.657678 24099 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 22:50:56.653327 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.147.195:45217 (local address 127.14.147.254:40759)
0929 22:50:56.653378 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0929 22:50:56.653379 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0929 22:50:56.653493 (+ 114us) server_negotiation.cc:407] Connection header received
0929 22:50:56.653538 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 22:50:56.653539 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 22:50:56.653549 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 22:50:56.653596 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0929 22:50:56.653909 (+ 313us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:50:56.654407 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:50:56.655111 (+ 704us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:50:56.655285 (+ 174us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:50:56.655327 (+ 42us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 22:50:56.657616 (+ 2289us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.147.195:45217: BlockingRecv error: recv got EOF from 127.14.147.195:45217 (error 108)
Metrics: {"server-negotiator.queue_time_us":27}
Sep 29 22:50:57 dist-test-slave-98mk krb5kdc[23064](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.147.193@KRBTEST.COM for oryx/127.14.147.254@KRBTEST.COM, Server not found in Kerberos database
Sep 29 22:50:57 dist-test-slave-98mk krb5kdc[23064](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.147.193@KRBTEST.COM for oryx/127.14.147.254@KRBTEST.COM, Server not found in Kerberos database
I20260929 22:50:57.392634 24102 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:50:57.387934 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.147.254:40759 (local address 127.14.147.193:33451)
0929 22:50:57.388100 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0929 22:50:57.388111 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0929 22:50:57.388167 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 22:50:57.388307 (+ 140us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 22:50:57.388307 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 22:50:57.388470 (+ 163us) client_negotiation.cc:189] Negotiated authn=SASL
0929 22:50:57.388518 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 22:50:57.388520 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 22:50:57.389189 (+ 669us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 22:50:57.389189 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 22:50:57.389827 (+ 638us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 22:50:57.389831 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 22:50:57.390179 (+ 348us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 22:50:57.390180 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 22:50:57.390229 (+ 49us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 22:50:57.390672 (+ 443us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 22:50:57.390675 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0929 22:50:57.392542 (+ 1867us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.147.254:40759: Server oryx/127.14.147.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":145,"thread_start_us":62,"threads_started":1}
W20260929 22:50:57.392707 24103 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 22:50:57.388070 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.147.193:33451 (local address 127.14.147.254:40759)
0929 22:50:57.388193 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0929 22:50:57.388194 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0929 22:50:57.388202 (+ 8us) server_negotiation.cc:407] Connection header received
0929 22:50:57.388227 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 22:50:57.388227 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 22:50:57.388237 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 22:50:57.388293 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0929 22:50:57.388609 (+ 316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:50:57.389119 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:50:57.389916 (+ 797us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:50:57.390120 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:50:57.390172 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 22:50:57.392644 (+ 2472us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.147.193:33451: BlockingRecv error: recv got EOF from 127.14.147.193:33451 (error 108)
Metrics: {"server-negotiator.queue_time_us":103,"thread_start_us":60,"threads_started":1}
Sep 29 22:50:57 dist-test-slave-98mk krb5kdc[23064](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.147.194@KRBTEST.COM for oryx/127.14.147.254@KRBTEST.COM, Server not found in Kerberos database
Sep 29 22:50:57 dist-test-slave-98mk krb5kdc[23064](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.147.194@KRBTEST.COM for oryx/127.14.147.254@KRBTEST.COM, Server not found in Kerberos database
I20260929 22:50:57.523777 24104 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:50:57.519237 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.147.254:40759 (local address 127.14.147.194:44867)
0929 22:50:57.519375 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0929 22:50:57.519394 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0929 22:50:57.519476 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 22:50:57.519614 (+ 138us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 22:50:57.519614 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 22:50:57.519770 (+ 156us) client_negotiation.cc:189] Negotiated authn=SASL
0929 22:50:57.519830 (+ 60us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 22:50:57.519832 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 22:50:57.520594 (+ 762us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 22:50:57.520594 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 22:50:57.521121 (+ 527us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 22:50:57.521123 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 22:50:57.521413 (+ 290us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 22:50:57.521414 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 22:50:57.521453 (+ 39us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 22:50:57.521861 (+ 408us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 22:50:57.521867 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0929 22:50:57.523684 (+ 1817us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.147.254:40759: Server oryx/127.14.147.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":66,"threads_started":1}
W20260929 22:50:57.523839 24103 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 22:50:57.519345 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.147.194:44867 (local address 127.14.147.254:40759)
0929 22:50:57.519408 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0929 22:50:57.519410 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 22:50:57.519460 (+ 50us) server_negotiation.cc:407] Connection header received
0929 22:50:57.519529 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 22:50:57.519530 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 22:50:57.519543 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 22:50:57.519601 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0929 22:50:57.519882 (+ 281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:50:57.520532 (+ 650us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:50:57.521179 (+ 647us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:50:57.521363 (+ 184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:50:57.521406 (+ 43us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 22:50:57.523787 (+ 2381us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.147.194:44867: BlockingRecv error: recv got EOF from 127.14.147.194:44867 (error 108)
Metrics: {"server-negotiator.queue_time_us":31}
Sep 29 22:50:57 dist-test-slave-98mk krb5kdc[23064](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.147.195@KRBTEST.COM for oryx/127.14.147.254@KRBTEST.COM, Server not found in Kerberos database
Sep 29 22:50:57 dist-test-slave-98mk krb5kdc[23064](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.147.195@KRBTEST.COM for oryx/127.14.147.254@KRBTEST.COM, Server not found in Kerberos database
I20260929 22:50:57.662328 24105 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:50:57.658120 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.147.254:40759 (local address 127.14.147.195:40581)
0929 22:50:57.658244 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0929 22:50:57.658255 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0929 22:50:57.658303 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 22:50:57.658457 (+ 154us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 22:50:57.658458 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 22:50:57.658606 (+ 148us) client_negotiation.cc:189] Negotiated authn=SASL
0929 22:50:57.658657 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 22:50:57.658659 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 22:50:57.659253 (+ 594us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 22:50:57.659254 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 22:50:57.659802 (+ 548us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 22:50:57.659804 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 22:50:57.660147 (+ 343us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 22:50:57.660148 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 22:50:57.660200 (+ 52us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 22:50:57.660620 (+ 420us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 22:50:57.660624 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0929 22:50:57.662241 (+ 1617us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.147.254:40759: Server oryx/127.14.147.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":103,"thread_start_us":66,"threads_started":1}
W20260929 22:50:57.662480 24103 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 22:50:57.658243 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.147.195:40581 (local address 127.14.147.254:40759)
0929 22:50:57.658327 (+ 84us) server_negotiation.cc:206] Beginning negotiation
0929 22:50:57.658329 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 22:50:57.658340 (+ 11us) server_negotiation.cc:407] Connection header received
0929 22:50:57.658364 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 22:50:57.658364 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 22:50:57.658376 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 22:50:57.658443 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0929 22:50:57.658713 (+ 270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:50:57.659200 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:50:57.659861 (+ 661us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:50:57.660044 (+ 183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:50:57.660095 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 22:50:57.662353 (+ 2258us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.147.195:40581: BlockingRecv error: recv got EOF from 127.14.147.195:40581 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
Sep 29 22:50:58 dist-test-slave-98mk krb5kdc[23064](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.147.193@KRBTEST.COM for oryx/127.14.147.254@KRBTEST.COM, Server not found in Kerberos database
Sep 29 22:50:58 dist-test-slave-98mk krb5kdc[23064](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.147.193@KRBTEST.COM for oryx/127.14.147.254@KRBTEST.COM, Server not found in Kerberos database
I20260929 22:50:58.398023 24106 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:50:58.393243 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.147.254:40759 (local address 127.14.147.193:38017)
0929 22:50:58.393423 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0929 22:50:58.393433 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0929 22:50:58.393483 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 22:50:58.393626 (+ 143us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 22:50:58.393627 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 22:50:58.393793 (+ 166us) client_negotiation.cc:189] Negotiated authn=SASL
0929 22:50:58.393845 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 22:50:58.393847 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 22:50:58.394561 (+ 714us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 22:50:58.394562 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 22:50:58.395120 (+ 558us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 22:50:58.395122 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 22:50:58.395528 (+ 406us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 22:50:58.395529 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 22:50:58.395569 (+ 40us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 22:50:58.396000 (+ 431us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 22:50:58.396003 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0929 22:50:58.397943 (+ 1940us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.147.254:40759: Server oryx/127.14.147.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":160,"thread_start_us":91,"threads_started":1}
W20260929 22:50:58.398116 24107 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 22:50:58.393373 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.147.193:38017 (local address 127.14.147.254:40759)
0929 22:50:58.393496 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0929 22:50:58.393498 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 22:50:58.393507 (+ 9us) server_negotiation.cc:407] Connection header received
0929 22:50:58.393532 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 22:50:58.393532 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 22:50:58.393543 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 22:50:58.393613 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0929 22:50:58.393947 (+ 334us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:50:58.394470 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:50:58.395218 (+ 748us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:50:58.395463 (+ 245us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:50:58.395523 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 22:50:58.398052 (+ 2529us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.147.193:38017: BlockingRecv error: recv got EOF from 127.14.147.193:38017 (error 108)
Metrics: {"server-negotiator.queue_time_us":102,"thread_start_us":59,"threads_started":1}
Sep 29 22:50:58 dist-test-slave-98mk krb5kdc[23064](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.147.194@KRBTEST.COM for oryx/127.14.147.254@KRBTEST.COM, Server not found in Kerberos database
Sep 29 22:50:58 dist-test-slave-98mk krb5kdc[23064](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.147.194@KRBTEST.COM for oryx/127.14.147.254@KRBTEST.COM, Server not found in Kerberos database
I20260929 22:50:58.528721 24108 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:50:58.524349 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.147.254:40759 (local address 127.14.147.194:46205)
0929 22:50:58.524481 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0929 22:50:58.524494 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 22:50:58.524570 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 22:50:58.524689 (+ 119us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 22:50:58.524690 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 22:50:58.524850 (+ 160us) client_negotiation.cc:189] Negotiated authn=SASL
0929 22:50:58.524905 (+ 55us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 22:50:58.524907 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 22:50:58.525497 (+ 590us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 22:50:58.525498 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 22:50:58.526016 (+ 518us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 22:50:58.526018 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 22:50:58.526384 (+ 366us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 22:50:58.526385 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 22:50:58.526439 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 22:50:58.526947 (+ 508us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 22:50:58.526951 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0929 22:50:58.528659 (+ 1708us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.147.254:40759: Server oryx/127.14.147.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":61,"threads_started":1}
W20260929 22:50:58.528831 24107 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 22:50:58.524473 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.147.194:46205 (local address 127.14.147.254:40759)
0929 22:50:58.524540 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0929 22:50:58.524542 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 22:50:58.524552 (+ 10us) server_negotiation.cc:407] Connection header received
0929 22:50:58.524627 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 22:50:58.524628 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 22:50:58.524638 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 22:50:58.524683 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0929 22:50:58.524953 (+ 270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:50:58.525441 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:50:58.526095 (+ 654us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:50:58.526288 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:50:58.526339 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 22:50:58.528757 (+ 2418us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.147.194:46205: BlockingRecv error: recv got EOF from 127.14.147.194:46205 (error 108)
Metrics: {"server-negotiator.queue_time_us":41}
Sep 29 22:50:58 dist-test-slave-98mk krb5kdc[23064](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.147.195@KRBTEST.COM for oryx/127.14.147.254@KRBTEST.COM, Server not found in Kerberos database
Sep 29 22:50:58 dist-test-slave-98mk krb5kdc[23064](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.147.195@KRBTEST.COM for oryx/127.14.147.254@KRBTEST.COM, Server not found in Kerberos database
I20260929 22:50:58.670737 24109 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:50:58.662776 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.147.254:40759 (local address 127.14.147.195:43749)
0929 22:50:58.662985 (+ 209us) negotiation.cc:107] Waiting for socket to connect
0929 22:50:58.662997 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0929 22:50:58.663075 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 22:50:58.663215 (+ 140us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 22:50:58.663216 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 22:50:58.663403 (+ 187us) client_negotiation.cc:189] Negotiated authn=SASL
0929 22:50:58.663469 (+ 66us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 22:50:58.663471 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 22:50:58.664130 (+ 659us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 22:50:58.664132 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 22:50:58.664651 (+ 519us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 22:50:58.664653 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 22:50:58.664993 (+ 340us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 22:50:58.664994 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 22:50:58.665047 (+ 53us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 22:50:58.668826 (+ 3779us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 22:50:58.668832 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0929 22:50:58.670614 (+ 1782us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.147.254:40759: Server oryx/127.14.147.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":193,"thread_start_us":81,"threads_started":1}
W20260929 22:50:58.670800 24107 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 22:50:58.662871 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.147.195:43749 (local address 127.14.147.254:40759)
0929 22:50:58.662934 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0929 22:50:58.662937 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 22:50:58.663068 (+ 131us) server_negotiation.cc:407] Connection header received
0929 22:50:58.663137 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 22:50:58.663138 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 22:50:58.663152 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 22:50:58.663199 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0929 22:50:58.663559 (+ 360us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:50:58.664052 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:50:58.664738 (+ 686us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:50:58.664927 (+ 189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:50:58.664982 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 22:50:58.670738 (+ 5756us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.147.195:43749: BlockingRecv error: recv got EOF from 127.14.147.195:43749 (error 108)
Metrics: {"server-negotiator.queue_time_us":30}
Sep 29 22:50:59 dist-test-slave-98mk krb5kdc[23064](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.147.193@KRBTEST.COM for oryx/127.14.147.254@KRBTEST.COM, Server not found in Kerberos database
Sep 29 22:50:59 dist-test-slave-98mk krb5kdc[23064](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.147.193@KRBTEST.COM for oryx/127.14.147.254@KRBTEST.COM, Server not found in Kerberos database
I20260929 22:50:59.402889 24110 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:50:59.398465 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.147.254:40759 (local address 127.14.147.193:53703)
0929 22:50:59.398588 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0929 22:50:59.398600 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0929 22:50:59.398648 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 22:50:59.398764 (+ 116us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 22:50:59.398765 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 22:50:59.398920 (+ 155us) client_negotiation.cc:189] Negotiated authn=SASL
0929 22:50:59.398969 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 22:50:59.398971 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 22:50:59.399694 (+ 723us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 22:50:59.399694 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 22:50:59.400210 (+ 516us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 22:50:59.400211 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 22:50:59.400556 (+ 345us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 22:50:59.400557 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 22:50:59.400593 (+ 36us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 22:50:59.401006 (+ 413us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 22:50:59.401010 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0929 22:50:59.402808 (+ 1798us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.147.254:40759: Server oryx/127.14.147.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":103,"thread_start_us":69,"threads_started":1}
W20260929 22:50:59.402976 24111 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 22:50:59.398530 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.147.193:53703 (local address 127.14.147.254:40759)
0929 22:50:59.398632 (+ 102us) server_negotiation.cc:206] Beginning negotiation
0929 22:50:59.398634 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 22:50:59.398642 (+ 8us) server_negotiation.cc:407] Connection header received
0929 22:50:59.398692 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 22:50:59.398693 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 22:50:59.398703 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 22:50:59.398754 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0929 22:50:59.399060 (+ 306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:50:59.399629 (+ 569us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:50:59.400293 (+ 664us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:50:59.400493 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:50:59.400552 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 22:50:59.402913 (+ 2361us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.147.193:53703: BlockingRecv error: recv got EOF from 127.14.147.193:53703 (error 108)
Metrics: {"server-negotiator.queue_time_us":81,"thread_start_us":51,"threads_started":1}
Sep 29 22:50:59 dist-test-slave-98mk krb5kdc[23064](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.147.194@KRBTEST.COM for oryx/127.14.147.254@KRBTEST.COM, Server not found in Kerberos database
Sep 29 22:50:59 dist-test-slave-98mk krb5kdc[23064](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.147.194@KRBTEST.COM for oryx/127.14.147.254@KRBTEST.COM, Server not found in Kerberos database
I20260929 22:50:59.533726 24112 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:50:59.529243 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.147.254:40759 (local address 127.14.147.194:42583)
0929 22:50:59.529365 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0929 22:50:59.529376 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0929 22:50:59.529439 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 22:50:59.529588 (+ 149us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 22:50:59.529589 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 22:50:59.529744 (+ 155us) client_negotiation.cc:189] Negotiated authn=SASL
0929 22:50:59.529792 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 22:50:59.529793 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 22:50:59.530467 (+ 674us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 22:50:59.530467 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 22:50:59.531002 (+ 535us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 22:50:59.531004 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 22:50:59.531309 (+ 305us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 22:50:59.531310 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 22:50:59.531367 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 22:50:59.531834 (+ 467us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 22:50:59.531839 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0929 22:50:59.533635 (+ 1796us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.147.254:40759: Server oryx/127.14.147.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":103,"thread_start_us":65,"threads_started":1}
W20260929 22:50:59.533808 24111 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 22:50:59.529326 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.147.194:42583 (local address 127.14.147.254:40759)
0929 22:50:59.529396 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0929 22:50:59.529398 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 22:50:59.529437 (+ 39us) server_negotiation.cc:407] Connection header received
0929 22:50:59.529502 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 22:50:59.529503 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 22:50:59.529513 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 22:50:59.529570 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0929 22:50:59.529884 (+ 314us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:50:59.530397 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:50:59.531061 (+ 664us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:50:59.531242 (+ 181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:50:59.531285 (+ 43us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 22:50:59.533742 (+ 2457us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.147.194:42583: BlockingRecv error: recv got EOF from 127.14.147.194:42583 (error 108)
Metrics: {"server-negotiator.queue_time_us":45}
Sep 29 22:50:59 dist-test-slave-98mk krb5kdc[23064](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.147.195@KRBTEST.COM for oryx/127.14.147.254@KRBTEST.COM, Server not found in Kerberos database
Sep 29 22:50:59 dist-test-slave-98mk krb5kdc[23064](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.147.195@KRBTEST.COM for oryx/127.14.147.254@KRBTEST.COM, Server not found in Kerberos database
I20260929 22:50:59.676126 24113 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:50:59.671335 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.147.254:40759 (local address 127.14.147.195:52325)
0929 22:50:59.671483 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0929 22:50:59.671495 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0929 22:50:59.671552 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 22:50:59.671672 (+ 120us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 22:50:59.671673 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 22:50:59.671834 (+ 161us) client_negotiation.cc:189] Negotiated authn=SASL
0929 22:50:59.671887 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 22:50:59.671889 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 22:50:59.672877 (+ 988us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 22:50:59.672878 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 22:50:59.673419 (+ 541us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 22:50:59.673421 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 22:50:59.673763 (+ 342us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 22:50:59.673763 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 22:50:59.673837 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 22:50:59.674305 (+ 468us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 22:50:59.674310 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0929 22:50:59.676064 (+ 1754us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.147.254:40759: Server oryx/127.14.147.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":59,"threads_started":1}
W20260929 22:50:59.676220 24111 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 22:50:59.671468 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.147.195:52325 (local address 127.14.147.254:40759)
0929 22:50:59.671526 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0929 22:50:59.671528 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 22:50:59.671540 (+ 12us) server_negotiation.cc:407] Connection header received
0929 22:50:59.671601 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 22:50:59.671602 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 22:50:59.671614 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 22:50:59.671659 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0929 22:50:59.672038 (+ 379us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:50:59.672806 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:50:59.673500 (+ 694us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:50:59.673705 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:50:59.673761 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 22:50:59.676159 (+ 2398us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.147.195:52325: BlockingRecv error: recv got EOF from 127.14.147.195:52325 (error 108)
Metrics: {"server-negotiator.queue_time_us":33}
Sep 29 22:51:00 dist-test-slave-98mk krb5kdc[23064](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.147.193@KRBTEST.COM for oryx/127.14.147.254@KRBTEST.COM, Server not found in Kerberos database
Sep 29 22:51:00 dist-test-slave-98mk krb5kdc[23064](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.147.193@KRBTEST.COM for oryx/127.14.147.254@KRBTEST.COM, Server not found in Kerberos database
I20260929 22:51:00.408043 24114 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:51:00.403453 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.147.254:40759 (local address 127.14.147.193:44845)
0929 22:51:00.403583 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0929 22:51:00.403594 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0929 22:51:00.403661 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 22:51:00.403786 (+ 125us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 22:51:00.403787 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 22:51:00.403948 (+ 161us) client_negotiation.cc:189] Negotiated authn=SASL
0929 22:51:00.403995 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 22:51:00.403996 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 22:51:00.404614 (+ 618us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 22:51:00.404615 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 22:51:00.405157 (+ 542us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 22:51:00.405159 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 22:51:00.405535 (+ 376us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 22:51:00.405535 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 22:51:00.405574 (+ 39us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 22:51:00.406041 (+ 467us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 22:51:00.406045 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0929 22:51:00.407968 (+ 1923us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.147.254:40759: Server oryx/127.14.147.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":72,"threads_started":1}
W20260929 22:51:00.408205 24116 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 22:51:00.403511 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.147.193:44845 (local address 127.14.147.254:40759)
0929 22:51:00.403612 (+ 101us) server_negotiation.cc:206] Beginning negotiation
0929 22:51:00.403613 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0929 22:51:00.403657 (+ 44us) server_negotiation.cc:407] Connection header received
0929 22:51:00.403706 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 22:51:00.403707 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 22:51:00.403718 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 22:51:00.403774 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0929 22:51:00.404047 (+ 273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:51:00.404557 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:51:00.405263 (+ 706us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:51:00.405469 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:51:00.405532 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 22:51:00.408123 (+ 2591us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.147.193:44845: BlockingRecv error: recv got EOF from 127.14.147.193:44845 (error 108)
Metrics: {"server-negotiator.queue_time_us":81,"thread_start_us":48,"threads_started":1}
Sep 29 22:51:00 dist-test-slave-98mk krb5kdc[23064](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.147.194@KRBTEST.COM for oryx/127.14.147.254@KRBTEST.COM, Server not found in Kerberos database
Sep 29 22:51:00 dist-test-slave-98mk krb5kdc[23064](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.147.194@KRBTEST.COM for oryx/127.14.147.254@KRBTEST.COM, Server not found in Kerberos database
I20260929 22:51:00.539021 24117 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:51:00.534367 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.147.254:40759 (local address 127.14.147.194:38011)
0929 22:51:00.534521 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0929 22:51:00.534534 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 22:51:00.534607 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 22:51:00.534740 (+ 133us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 22:51:00.534741 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 22:51:00.534920 (+ 179us) client_negotiation.cc:189] Negotiated authn=SASL
0929 22:51:00.534965 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 22:51:00.534967 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 22:51:00.535668 (+ 701us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 22:51:00.535669 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 22:51:00.536278 (+ 609us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 22:51:00.536281 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 22:51:00.536619 (+ 338us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 22:51:00.536620 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 22:51:00.536675 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 22:51:00.537109 (+ 434us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 22:51:00.537113 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0929 22:51:00.538950 (+ 1837us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.147.254:40759: Server oryx/127.14.147.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":62,"threads_started":1}
W20260929 22:51:00.539143 24116 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 22:51:00.534486 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.147.194:38011 (local address 127.14.147.254:40759)
0929 22:51:00.534569 (+ 83us) server_negotiation.cc:206] Beginning negotiation
0929 22:51:00.534571 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 22:51:00.534585 (+ 14us) server_negotiation.cc:407] Connection header received
0929 22:51:00.534673 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 22:51:00.534674 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 22:51:00.534687 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 22:51:00.534735 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0929 22:51:00.535039 (+ 304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:51:00.535598 (+ 559us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:51:00.536354 (+ 756us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:51:00.536559 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:51:00.536613 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 22:51:00.539071 (+ 2458us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.147.194:38011: BlockingRecv error: recv got EOF from 127.14.147.194:38011 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
Sep 29 22:51:00 dist-test-slave-98mk krb5kdc[23064](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.147.195@KRBTEST.COM for oryx/127.14.147.254@KRBTEST.COM, Server not found in Kerberos database
Sep 29 22:51:00 dist-test-slave-98mk krb5kdc[23064](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.147.195@KRBTEST.COM for oryx/127.14.147.254@KRBTEST.COM, Server not found in Kerberos database
I20260929 22:51:00.681486 24118 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:51:00.676586 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.147.254:40759 (local address 127.14.147.195:55711)
0929 22:51:00.676728 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0929 22:51:00.676739 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0929 22:51:00.676795 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 22:51:00.676946 (+ 151us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 22:51:00.676947 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 22:51:00.677122 (+ 175us) client_negotiation.cc:189] Negotiated authn=SASL
0929 22:51:00.677168 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 22:51:00.677170 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 22:51:00.677803 (+ 633us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 22:51:00.677803 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 22:51:00.678344 (+ 541us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 22:51:00.678345 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 22:51:00.678658 (+ 313us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 22:51:00.678659 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 22:51:00.678711 (+ 52us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 22:51:00.679238 (+ 527us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 22:51:00.679242 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0929 22:51:00.681378 (+ 2136us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.147.254:40759: Server oryx/127.14.147.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":64,"threads_started":1}
W20260929 22:51:00.681550 24116 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 22:51:00.676755 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.147.195:55711 (local address 127.14.147.254:40759)
0929 22:51:00.676837 (+ 82us) server_negotiation.cc:206] Beginning negotiation
0929 22:51:00.676840 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 22:51:00.676848 (+ 8us) server_negotiation.cc:407] Connection header received
0929 22:51:00.676871 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 22:51:00.676872 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 22:51:00.676886 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 22:51:00.676939 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0929 22:51:00.677235 (+ 296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:51:00.677730 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:51:00.678412 (+ 682us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:51:00.678600 (+ 188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:51:00.678654 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 22:51:00.681495 (+ 2841us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.147.195:55711: BlockingRecv error: recv got EOF from 127.14.147.195:55711 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
Sep 29 22:51:01 dist-test-slave-98mk krb5kdc[23064](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.147.193@KRBTEST.COM for oryx/127.14.147.254@KRBTEST.COM, Server not found in Kerberos database
Sep 29 22:51:01 dist-test-slave-98mk krb5kdc[23064](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.147.193@KRBTEST.COM for oryx/127.14.147.254@KRBTEST.COM, Server not found in Kerberos database
I20260929 22:51:01.413282 24119 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:51:01.408635 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.147.254:40759 (local address 127.14.147.193:59979)
0929 22:51:01.408767 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0929 22:51:01.408779 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0929 22:51:01.408850 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 22:51:01.409104 (+ 254us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 22:51:01.409106 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 22:51:01.409320 (+ 214us) client_negotiation.cc:189] Negotiated authn=SASL
0929 22:51:01.409382 (+ 62us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 22:51:01.409385 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 22:51:01.410030 (+ 645us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 22:51:01.410030 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 22:51:01.410569 (+ 539us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 22:51:01.410571 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 22:51:01.410872 (+ 301us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 22:51:01.410873 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 22:51:01.410909 (+ 36us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 22:51:01.411360 (+ 451us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 22:51:01.411365 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0929 22:51:01.413175 (+ 1810us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.147.254:40759: Server oryx/127.14.147.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":69,"threads_started":1}
W20260929 22:51:01.413322 24120 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 22:51:01.408779 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.147.193:59979 (local address 127.14.147.254:40759)
0929 22:51:01.408895 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0929 22:51:01.408897 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 22:51:01.408907 (+ 10us) server_negotiation.cc:407] Connection header received
0929 22:51:01.408937 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 22:51:01.408938 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 22:51:01.408948 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 22:51:01.409017 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0929 22:51:01.409461 (+ 444us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:51:01.409970 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:51:01.410637 (+ 667us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:51:01.410825 (+ 188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:51:01.410868 (+ 43us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 22:51:01.413259 (+ 2391us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.147.193:59979: BlockingRecv error: recv got EOF from 127.14.147.193:59979 (error 108)
Metrics: {"server-negotiator.queue_time_us":95,"thread_start_us":48,"threads_started":1}
Sep 29 22:51:01 dist-test-slave-98mk krb5kdc[23064](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.147.194@KRBTEST.COM for oryx/127.14.147.254@KRBTEST.COM, Server not found in Kerberos database
Sep 29 22:51:01 dist-test-slave-98mk krb5kdc[23064](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.147.194@KRBTEST.COM for oryx/127.14.147.254@KRBTEST.COM, Server not found in Kerberos database
I20260929 22:51:01.544013 24121 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:51:01.539535 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.147.254:40759 (local address 127.14.147.194:35359)
0929 22:51:01.539646 (+ 111us) negotiation.cc:107] Waiting for socket to connect
0929 22:51:01.539656 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0929 22:51:01.539750 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 22:51:01.539885 (+ 135us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 22:51:01.539885 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 22:51:01.540052 (+ 167us) client_negotiation.cc:189] Negotiated authn=SASL
0929 22:51:01.540097 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 22:51:01.540099 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 22:51:01.540782 (+ 683us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 22:51:01.540783 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 22:51:01.541501 (+ 718us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 22:51:01.541504 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 22:51:01.541816 (+ 312us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 22:51:01.541816 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 22:51:01.541853 (+ 37us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 22:51:01.542221 (+ 368us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 22:51:01.542225 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0929 22:51:01.543952 (+ 1727us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.147.254:40759: Server oryx/127.14.147.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":90,"thread_start_us":61,"threads_started":1}
W20260929 22:51:01.544106 24120 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 22:51:01.539580 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.147.194:35359 (local address 127.14.147.254:40759)
0929 22:51:01.539654 (+ 74us) server_negotiation.cc:206] Beginning negotiation
0929 22:51:01.539656 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 22:51:01.539734 (+ 78us) server_negotiation.cc:407] Connection header received
0929 22:51:01.539818 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 22:51:01.539819 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 22:51:01.539831 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 22:51:01.539879 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0929 22:51:01.540163 (+ 284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:51:01.540670 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:51:01.541563 (+ 893us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:51:01.541754 (+ 191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:51:01.541807 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 22:51:01.544044 (+ 2237us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.147.194:35359: BlockingRecv error: recv got EOF from 127.14.147.194:35359 (error 108)
Metrics: {"server-negotiator.queue_time_us":44}
Sep 29 22:51:01 dist-test-slave-98mk krb5kdc[23064](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.147.195@KRBTEST.COM for oryx/127.14.147.254@KRBTEST.COM, Server not found in Kerberos database
Sep 29 22:51:01 dist-test-slave-98mk krb5kdc[23064](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.147.195@KRBTEST.COM for oryx/127.14.147.254@KRBTEST.COM, Server not found in Kerberos database
I20260929 22:51:01.686631 24122 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:51:01.682129 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.147.254:40759 (local address 127.14.147.195:60507)
0929 22:51:01.682280 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0929 22:51:01.682290 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0929 22:51:01.682345 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 22:51:01.682497 (+ 152us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 22:51:01.682498 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 22:51:01.682674 (+ 176us) client_negotiation.cc:189] Negotiated authn=SASL
0929 22:51:01.682722 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 22:51:01.682724 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 22:51:01.683356 (+ 632us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 22:51:01.683357 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 22:51:01.683903 (+ 546us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 22:51:01.683904 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 22:51:01.684226 (+ 322us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 22:51:01.684227 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 22:51:01.684277 (+ 50us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 22:51:01.684728 (+ 451us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 22:51:01.684732 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0929 22:51:01.686569 (+ 1837us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.147.254:40759: Server oryx/127.14.147.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":67,"threads_started":1}
W20260929 22:51:01.686723 24120 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 22:51:01.682301 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.147.195:60507 (local address 127.14.147.254:40759)
0929 22:51:01.682375 (+ 74us) server_negotiation.cc:206] Beginning negotiation
0929 22:51:01.682377 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 22:51:01.682387 (+ 10us) server_negotiation.cc:407] Connection header received
0929 22:51:01.682415 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 22:51:01.682416 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 22:51:01.682437 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 22:51:01.682481 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0929 22:51:01.682795 (+ 314us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:51:01.683282 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:51:01.683983 (+ 701us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:51:01.684166 (+ 183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:51:01.684222 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 22:51:01.686655 (+ 2433us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.147.195:60507: BlockingRecv error: recv got EOF from 127.14.147.195:60507 (error 108)
Metrics: {"server-negotiator.queue_time_us":47}
Sep 29 22:51:02 dist-test-slave-98mk krb5kdc[23064](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.147.193@KRBTEST.COM for oryx/127.14.147.254@KRBTEST.COM, Server not found in Kerberos database
Sep 29 22:51:02 dist-test-slave-98mk krb5kdc[23064](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.147.193@KRBTEST.COM for oryx/127.14.147.254@KRBTEST.COM, Server not found in Kerberos database
I20260929 22:51:02.418174 24123 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:51:02.413934 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.147.254:40759 (local address 127.14.147.193:56593)
0929 22:51:02.414097 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0929 22:51:02.414108 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0929 22:51:02.414159 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 22:51:02.414320 (+ 161us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 22:51:02.414320 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 22:51:02.414492 (+ 172us) client_negotiation.cc:189] Negotiated authn=SASL
0929 22:51:02.414538 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 22:51:02.414540 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 22:51:02.415148 (+ 608us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 22:51:02.415148 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 22:51:02.415709 (+ 561us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 22:51:02.415711 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 22:51:02.416045 (+ 334us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 22:51:02.416046 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 22:51:02.416084 (+ 38us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 22:51:02.416536 (+ 452us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 22:51:02.416540 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0929 22:51:02.418111 (+ 1571us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.147.254:40759: Server oryx/127.14.147.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":143,"thread_start_us":72,"threads_started":1}
W20260929 22:51:02.418289 24124 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 22:51:02.414053 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.147.193:56593 (local address 127.14.147.254:40759)
0929 22:51:02.414182 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0929 22:51:02.414184 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 22:51:02.414192 (+ 8us) server_negotiation.cc:407] Connection header received
0929 22:51:02.414215 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 22:51:02.414216 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 22:51:02.414226 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 22:51:02.414305 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0929 22:51:02.414598 (+ 293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:51:02.415092 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:51:02.415791 (+ 699us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:51:02.415987 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:51:02.416040 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 22:51:02.418214 (+ 2174us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.147.193:56593: BlockingRecv error: recv got EOF from 127.14.147.193:56593 (error 108)
Metrics: {"server-negotiator.queue_time_us":107,"thread_start_us":81,"threads_started":1}
Sep 29 22:51:02 dist-test-slave-98mk krb5kdc[23064](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.147.194@KRBTEST.COM for oryx/127.14.147.254@KRBTEST.COM, Server not found in Kerberos database
Sep 29 22:51:02 dist-test-slave-98mk krb5kdc[23064](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.147.194@KRBTEST.COM for oryx/127.14.147.254@KRBTEST.COM, Server not found in Kerberos database
I20260929 22:51:02.548925 24125 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:51:02.544448 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.147.254:40759 (local address 127.14.147.194:34093)
0929 22:51:02.544589 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0929 22:51:02.544603 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 22:51:02.544670 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 22:51:02.544841 (+ 171us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 22:51:02.544843 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 22:51:02.545015 (+ 172us) client_negotiation.cc:189] Negotiated authn=SASL
0929 22:51:02.545058 (+ 43us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 22:51:02.545060 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 22:51:02.545730 (+ 670us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 22:51:02.545731 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 22:51:02.546297 (+ 566us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 22:51:02.546299 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 22:51:02.546618 (+ 319us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 22:51:02.546619 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 22:51:02.546671 (+ 52us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 22:51:02.547094 (+ 423us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 22:51:02.547099 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0929 22:51:02.548828 (+ 1729us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.147.254:40759: Server oryx/127.14.147.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":57,"threads_started":1}
W20260929 22:51:02.548995 24124 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 22:51:02.544576 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.147.194:34093 (local address 127.14.147.254:40759)
0929 22:51:02.544661 (+ 85us) server_negotiation.cc:206] Beginning negotiation
0929 22:51:02.544663 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 22:51:02.544672 (+ 9us) server_negotiation.cc:407] Connection header received
0929 22:51:02.544763 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 22:51:02.544764 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 22:51:02.544774 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 22:51:02.544826 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0929 22:51:02.545131 (+ 305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:51:02.545657 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:51:02.546373 (+ 716us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:51:02.546560 (+ 187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:51:02.546612 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 22:51:02.548940 (+ 2328us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.147.194:34093: BlockingRecv error: recv got EOF from 127.14.147.194:34093 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
Sep 29 22:51:02 dist-test-slave-98mk krb5kdc[23064](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.147.195@KRBTEST.COM for oryx/127.14.147.254@KRBTEST.COM, Server not found in Kerberos database
Sep 29 22:51:02 dist-test-slave-98mk krb5kdc[23064](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.147.195@KRBTEST.COM for oryx/127.14.147.254@KRBTEST.COM, Server not found in Kerberos database
I20260929 22:51:02.691627 24126 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:51:02.687191 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.147.254:40759 (local address 127.14.147.195:60687)
0929 22:51:02.687376 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0929 22:51:02.687390 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 22:51:02.687485 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 22:51:02.687616 (+ 131us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 22:51:02.687617 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 22:51:02.687796 (+ 179us) client_negotiation.cc:189] Negotiated authn=SASL
0929 22:51:02.687843 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 22:51:02.687845 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 22:51:02.688488 (+ 643us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 22:51:02.688489 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 22:51:02.688996 (+ 507us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 22:51:02.688998 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 22:51:02.689329 (+ 331us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 22:51:02.689330 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 22:51:02.689382 (+ 52us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 22:51:02.689821 (+ 439us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 22:51:02.689825 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0929 22:51:02.691541 (+ 1716us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.147.254:40759: Server oryx/127.14.147.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":168,"thread_start_us":78,"threads_started":1}
W20260929 22:51:02.691686 24124 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 22:51:02.687310 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.147.195:60687 (local address 127.14.147.254:40759)
0929 22:51:02.687380 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0929 22:51:02.687383 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 22:51:02.687476 (+ 93us) server_negotiation.cc:407] Connection header received
0929 22:51:02.687542 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 22:51:02.687543 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 22:51:02.687557 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 22:51:02.687601 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0929 22:51:02.687918 (+ 317us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:51:02.688411 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:51:02.689076 (+ 665us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:51:02.689259 (+ 183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:51:02.689314 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 22:51:02.691633 (+ 2319us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.147.195:60687: BlockingRecv error: recv got EOF from 127.14.147.195:60687 (error 108)
Metrics: {"server-negotiator.queue_time_us":40}
Sep 29 22:51:03 dist-test-slave-98mk krb5kdc[23064](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.147.193@KRBTEST.COM for oryx/127.14.147.254@KRBTEST.COM, Server not found in Kerberos database
Sep 29 22:51:03 dist-test-slave-98mk krb5kdc[23064](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.147.193@KRBTEST.COM for oryx/127.14.147.254@KRBTEST.COM, Server not found in Kerberos database
I20260929 22:51:03.423084 24127 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:51:03.418596 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.147.254:40759 (local address 127.14.147.193:44979)
0929 22:51:03.418718 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0929 22:51:03.418730 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0929 22:51:03.418780 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 22:51:03.418896 (+ 116us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 22:51:03.418897 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 22:51:03.419053 (+ 156us) client_negotiation.cc:189] Negotiated authn=SASL
0929 22:51:03.419110 (+ 57us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 22:51:03.419112 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 22:51:03.419770 (+ 658us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 22:51:03.419771 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 22:51:03.420327 (+ 556us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 22:51:03.420329 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 22:51:03.420734 (+ 405us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 22:51:03.420735 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 22:51:03.420781 (+ 46us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 22:51:03.421206 (+ 425us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 22:51:03.421210 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0929 22:51:03.423011 (+ 1801us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.147.254:40759: Server oryx/127.14.147.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":103,"thread_start_us":54,"threads_started":1}
W20260929 22:51:03.423202 24128 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 22:51:03.418643 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.147.193:44979 (local address 127.14.147.254:40759)
0929 22:51:03.418771 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0929 22:51:03.418773 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 22:51:03.418780 (+ 7us) server_negotiation.cc:407] Connection header received
0929 22:51:03.418822 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 22:51:03.418823 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 22:51:03.418833 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 22:51:03.418885 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0929 22:51:03.419163 (+ 278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:51:03.419711 (+ 548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:51:03.420457 (+ 746us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:51:03.420654 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:51:03.420714 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 22:51:03.423119 (+ 2405us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.147.193:44979: BlockingRecv error: recv got EOF from 127.14.147.193:44979 (error 108)
Metrics: {"server-negotiator.queue_time_us":108,"thread_start_us":68,"threads_started":1}
Sep 29 22:51:03 dist-test-slave-98mk krb5kdc[23064](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.147.194@KRBTEST.COM for oryx/127.14.147.254@KRBTEST.COM, Server not found in Kerberos database
Sep 29 22:51:03 dist-test-slave-98mk krb5kdc[23064](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.147.194@KRBTEST.COM for oryx/127.14.147.254@KRBTEST.COM, Server not found in Kerberos database
I20260929 22:51:03.554654 24129 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:51:03.549449 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.147.254:40759 (local address 127.14.147.194:39057)
0929 22:51:03.549613 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0929 22:51:03.549625 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0929 22:51:03.549704 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 22:51:03.549858 (+ 154us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 22:51:03.549858 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 22:51:03.550057 (+ 199us) client_negotiation.cc:189] Negotiated authn=SASL
0929 22:51:03.550112 (+ 55us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 22:51:03.550114 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 22:51:03.550871 (+ 757us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 22:51:03.550872 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 22:51:03.551455 (+ 583us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 22:51:03.551457 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 22:51:03.552002 (+ 545us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 22:51:03.552002 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 22:51:03.552047 (+ 45us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 22:51:03.552539 (+ 492us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 22:51:03.552543 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0929 22:51:03.554592 (+ 2049us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.147.254:40759: Server oryx/127.14.147.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":144,"thread_start_us":68,"threads_started":1}
W20260929 22:51:03.554951 24128 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 22:51:03.549558 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.147.194:39057 (local address 127.14.147.254:40759)
0929 22:51:03.549613 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0929 22:51:03.549615 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 22:51:03.549689 (+ 74us) server_negotiation.cc:407] Connection header received
0929 22:51:03.549771 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 22:51:03.549772 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 22:51:03.549782 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 22:51:03.549840 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0929 22:51:03.550194 (+ 354us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:51:03.550775 (+ 581us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:51:03.551607 (+ 832us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:51:03.551882 (+ 275us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:51:03.552009 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 22:51:03.554873 (+ 2864us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.147.194:39057: BlockingRecv error: recv got EOF from 127.14.147.194:39057 (error 108)
Metrics: {"server-negotiator.queue_time_us":29}
Sep 29 22:51:03 dist-test-slave-98mk krb5kdc[23064](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.147.195@KRBTEST.COM for oryx/127.14.147.254@KRBTEST.COM, Server not found in Kerberos database
Sep 29 22:51:03 dist-test-slave-98mk krb5kdc[23064](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.147.195@KRBTEST.COM for oryx/127.14.147.254@KRBTEST.COM, Server not found in Kerberos database
I20260929 22:51:03.701520 24130 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:51:03.692607 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.147.254:40759 (local address 127.14.147.195:54061)
0929 22:51:03.692777 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0929 22:51:03.692789 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0929 22:51:03.692871 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 22:51:03.693010 (+ 139us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 22:51:03.693011 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 22:51:03.693186 (+ 175us) client_negotiation.cc:189] Negotiated authn=SASL
0929 22:51:03.693233 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 22:51:03.693235 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 22:51:03.693885 (+ 650us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 22:51:03.693886 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 22:51:03.694419 (+ 533us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 22:51:03.694421 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 22:51:03.694745 (+ 324us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 22:51:03.694746 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 22:51:03.694798 (+ 52us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 22:51:03.699596 (+ 4798us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 22:51:03.699602 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0929 22:51:03.701424 (+ 1822us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.147.254:40759: Server oryx/127.14.147.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":151,"thread_start_us":76,"threads_started":1}
W20260929 22:51:03.701610 24128 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 22:51:03.692663 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.147.195:54061 (local address 127.14.147.254:40759)
0929 22:51:03.692752 (+ 89us) server_negotiation.cc:206] Beginning negotiation
0929 22:51:03.692755 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 22:51:03.692861 (+ 106us) server_negotiation.cc:407] Connection header received
0929 22:51:03.692930 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 22:51:03.692931 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 22:51:03.692945 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 22:51:03.692992 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0929 22:51:03.693309 (+ 317us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:51:03.693803 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:51:03.694503 (+ 700us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:51:03.694685 (+ 182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:51:03.694740 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 22:51:03.701548 (+ 6808us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.147.195:54061: BlockingRecv error: recv got EOF from 127.14.147.195:54061 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
Sep 29 22:51:04 dist-test-slave-98mk krb5kdc[23064](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.147.193@KRBTEST.COM for oryx/127.14.147.254@KRBTEST.COM, Server not found in Kerberos database
Sep 29 22:51:04 dist-test-slave-98mk krb5kdc[23064](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.147.193@KRBTEST.COM for oryx/127.14.147.254@KRBTEST.COM, Server not found in Kerberos database
I20260929 22:51:04.428210 24131 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:51:04.423702 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.147.254:40759 (local address 127.14.147.193:58743)
0929 22:51:04.423873 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0929 22:51:04.423883 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0929 22:51:04.423933 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 22:51:04.424085 (+ 152us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 22:51:04.424086 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 22:51:04.424250 (+ 164us) client_negotiation.cc:189] Negotiated authn=SASL
0929 22:51:04.424300 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 22:51:04.424302 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 22:51:04.424935 (+ 633us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 22:51:04.424936 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 22:51:04.425474 (+ 538us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 22:51:04.425476 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 22:51:04.425785 (+ 309us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 22:51:04.425785 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 22:51:04.425829 (+ 44us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 22:51:04.426259 (+ 430us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 22:51:04.426264 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0929 22:51:04.428144 (+ 1880us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.147.254:40759: Server oryx/127.14.147.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":148,"thread_start_us":71,"threads_started":1}
W20260929 22:51:04.428344 24132 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 22:51:04.423820 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.147.193:58743 (local address 127.14.147.254:40759)
0929 22:51:04.423935 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0929 22:51:04.423936 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0929 22:51:04.423984 (+ 48us) server_negotiation.cc:407] Connection header received
0929 22:51:04.424009 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 22:51:04.424010 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 22:51:04.424020 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 22:51:04.424068 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0929 22:51:04.424359 (+ 291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:51:04.424863 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:51:04.425538 (+ 675us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:51:04.425724 (+ 186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:51:04.425779 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 22:51:04.428271 (+ 2492us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.147.193:58743: BlockingRecv error: recv got EOF from 127.14.147.193:58743 (error 108)
Metrics: {"server-negotiator.queue_time_us":95,"thread_start_us":55,"threads_started":1}
Sep 29 22:51:04 dist-test-slave-98mk krb5kdc[23064](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.147.194@KRBTEST.COM for oryx/127.14.147.254@KRBTEST.COM, Server not found in Kerberos database
Sep 29 22:51:04 dist-test-slave-98mk krb5kdc[23064](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.147.194@KRBTEST.COM for oryx/127.14.147.254@KRBTEST.COM, Server not found in Kerberos database
I20260929 22:51:04.560719 24133 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:51:04.555527 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.147.254:40759 (local address 127.14.147.194:34889)
0929 22:51:04.555697 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0929 22:51:04.555715 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0929 22:51:04.555812 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 22:51:04.555952 (+ 140us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 22:51:04.555952 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 22:51:04.556138 (+ 186us) client_negotiation.cc:189] Negotiated authn=SASL
0929 22:51:04.556187 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 22:51:04.556189 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 22:51:04.556844 (+ 655us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 22:51:04.556844 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 22:51:04.557370 (+ 526us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 22:51:04.557371 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 22:51:04.557712 (+ 341us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 22:51:04.557713 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 22:51:04.557767 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 22:51:04.558238 (+ 471us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 22:51:04.558241 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0929 22:51:04.560619 (+ 2378us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.147.254:40759: Server oryx/127.14.147.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":139,"thread_start_us":94,"threads_started":1}
W20260929 22:51:04.560803 24132 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 22:51:04.555653 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.147.194:34889 (local address 127.14.147.254:40759)
0929 22:51:04.555731 (+ 78us) server_negotiation.cc:206] Beginning negotiation
0929 22:51:04.555734 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 22:51:04.555792 (+ 58us) server_negotiation.cc:407] Connection header received
0929 22:51:04.555877 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 22:51:04.555878 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 22:51:04.555890 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 22:51:04.555935 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0929 22:51:04.556256 (+ 321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:51:04.556763 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:51:04.557444 (+ 681us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:51:04.557649 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:51:04.557704 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 22:51:04.560740 (+ 3036us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.147.194:34889: BlockingRecv error: recv got EOF from 127.14.147.194:34889 (error 108)
Metrics: {"server-negotiator.queue_time_us":47}
I20260929 22:51:04.614320 14927 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhWOCLk/build/release/bin/kudu with pid 23651
I20260929 22:51:04.618598 14927 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhWOCLk/build/release/bin/kudu with pid 23804
I20260929 22:51:04.623834 14927 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhWOCLk/build/release/bin/kudu with pid 23938
I20260929 22:51:04.627801 14927 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhWOCLk/build/release/bin/kudu with pid 23581
2026-09-29T22:51:04Z chronyd exiting
[ OK ] SecurityITest.TestMismatchingPrincipals (17382 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationInsecureCluster
2026-09-29T22:51:04Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T22:51:04Z Disabled control of system clock
I20260929 22:51:04.653967 14927 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhWOCLk/build/release/bin/kudu
/tmp/dist-test-taskhWOCLk/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790722213368738-14927-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790722213368738-14927-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790722213368738-14927-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790722213368738-14927-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.147.254:46397
--webserver_interface=127.14.147.254
--webserver_port=0
--builtin_ntp_servers=127.14.147.212:33649
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.147.254:46397
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260929 22:51:04.733834 24137 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 22:51:04.734064 24137 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 22:51:04.734084 24137 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 22:51:04.735342 24137 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 22:51:04.735395 24137 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 22:51:04.735407 24137 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 22:51:04.735446 24137 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 22:51:04.735462 24137 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 22:51:04.736914 24137 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.147.212:33649
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790722213368738-14927-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790722213368738-14927-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.14.147.254:46397
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.147.254:46397
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790722213368738-14927-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.14.147.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24137
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790722213368738-14927-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 21:58:54 UTC on ccycloud.quasar-xguoti.root.comops.site
build id 4681390
I20260929 22:51:04.737183 24137 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 22:51:04.737481 24137 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 22:51:04.742890 24142 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 22:51:04.742937 24137 server_base.cc:1034] running on GCE node
W20260929 22:51:04.742882 24145 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 22:51:04.742882 24143 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 22:51:04.743311 24137 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 22:51:04.743652 24137 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 22:51:04.744843 24137 hybrid_clock.cc:648] HybridClock initialized: now 1790722264744810 us; error 34 us; skew 500 ppm
I20260929 22:51:04.746125 24137 webserver.cc:460] Webserver started at http://127.14.147.254:40931/ using document root <none> and password file <none>
I20260929 22:51:04.746337 24137 fs_manager.cc:359] Metadata directory not provided
I20260929 22:51:04.746380 24137 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 22:51:04.746474 24137 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 22:51:04.747557 24137 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790722213368738-14927-0/minicluster-data/master-0/data/instance:
uuid: "360bd4704b1148ec8fb3f750af7e33aa"
format_stamp: "Formatted at 2026-09-29 22:51:04 on dist-test-slave-98mk"
server_key: "41805c02cc3cc7ddf8c12e449697d0c0"
server_key_iv: "7077e61f3c9b85edc86e3f34acab2168"
server_key_version: "encryptionkey@0"
I20260929 22:51:04.747877 24137 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790722213368738-14927-0/minicluster-data/master-0/wal/instance:
uuid: "360bd4704b1148ec8fb3f750af7e33aa"
format_stamp: "Formatted at 2026-09-29 22:51:04 on dist-test-slave-98mk"
server_key: "41805c02cc3cc7ddf8c12e449697d0c0"
server_key_iv: "7077e61f3c9b85edc86e3f34acab2168"
server_key_version: "encryptionkey@0"
I20260929 22:51:04.749213 24137 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260929 22:51:04.750036 24151 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 22:51:04.750257 24137 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260929 22:51:04.750339 24137 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790722213368738-14927-0/minicluster-data/master-0/data,/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790722213368738-14927-0/minicluster-data/master-0/wal
uuid: "360bd4704b1148ec8fb3f750af7e33aa"
format_stamp: "Formatted at 2026-09-29 22:51:04 on dist-test-slave-98mk"
server_key: "41805c02cc3cc7ddf8c12e449697d0c0"
server_key_iv: "7077e61f3c9b85edc86e3f34acab2168"
server_key_version: "encryptionkey@0"
I20260929 22:51:04.750401 24137 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790722213368738-14927-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790722213368738-14927-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790722213368738-14927-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 22:51:04.782816 24137 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 22:51:04.783118 24137 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 22:51:04.783221 24137 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 22:51:04.786780 24137 rpc_server.cc:307] RPC server started. Bound to: 127.14.147.254:46397
I20260929 22:51:04.786816 24203 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.147.254:46397 every 8 connection(s)
I20260929 22:51:04.787158 24137 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790722213368738-14927-0/minicluster-data/master-0/data/info.pb
I20260929 22:51:04.787838 24204 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 22:51:04.788883 14927 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhWOCLk/build/release/bin/kudu as pid 24137
I20260929 22:51:04.788983 14927 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790722213368738-14927-0/minicluster-data/master-0/wal/instance
I20260929 22:51:04.789177 14927 external_mini_cluster.cc:1428] Setting key 6baa7628e616edf7d2eb046ebcbdfaea
I20260929 22:51:04.791119 24204 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 360bd4704b1148ec8fb3f750af7e33aa: Bootstrap starting.
I20260929 22:51:04.791757 24204 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 360bd4704b1148ec8fb3f750af7e33aa: Neither blocks nor log segments found. Creating new log.
I20260929 22:51:04.792049 24204 log.cc:824] T 00000000000000000000000000000000 P 360bd4704b1148ec8fb3f750af7e33aa: Log is configured to *not* fsync() on all Append() calls
I20260929 22:51:04.793007 24204 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 360bd4704b1148ec8fb3f750af7e33aa: No bootstrap required, opened a new log
I20260929 22:51:04.794762 24204 raft_consensus.cc:348] T 00000000000000000000000000000000 P 360bd4704b1148ec8fb3f750af7e33aa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "360bd4704b1148ec8fb3f750af7e33aa" member_type: VOTER last_known_addr { host: "127.14.147.254" port: 46397 } }
I20260929 22:51:04.794945 24204 raft_consensus.cc:374] T 00000000000000000000000000000000 P 360bd4704b1148ec8fb3f750af7e33aa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 22:51:04.795010 24204 raft_consensus.cc:729] T 00000000000000000000000000000000 P 360bd4704b1148ec8fb3f750af7e33aa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 360bd4704b1148ec8fb3f750af7e33aa, State: Initialized, Role: FOLLOWER
I20260929 22:51:04.795142 24204 consensus_queue.cc:260] T 00000000000000000000000000000000 P 360bd4704b1148ec8fb3f750af7e33aa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "360bd4704b1148ec8fb3f750af7e33aa" member_type: VOTER last_known_addr { host: "127.14.147.254" port: 46397 } }
I20260929 22:51:04.795408 24207 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:51:04.789719 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39136 (local address 127.14.147.254:46397)
0929 22:51:04.789896 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0929 22:51:04.789903 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 22:51:04.789927 (+ 24us) server_negotiation.cc:407] Connection header received
0929 22:51:04.790365 (+ 438us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 22:51:04.790372 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 22:51:04.790434 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 22:51:04.790543 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0929 22:51:04.790760 (+ 217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:51:04.791683 (+ 923us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:51:04.792659 (+ 976us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:51:04.793344 (+ 685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:51:04.793450 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 22:51:04.794227 (+ 777us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 22:51:04.794233 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 22:51:04.794233 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 22:51:04.794462 (+ 229us) server_negotiation.cc:922] calling sasl_server_start()
0929 22:51:04.794512 (+ 50us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 22:51:04.794529 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 22:51:04.794616 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0929 22:51:04.794726 (+ 110us) server_negotiation.cc:299] Negotiation successful
0929 22:51:04.794837 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":110,"thread_start_us":61,"threads_started":1}
I20260929 22:51:04.795462 24204 raft_consensus.cc:388] T 00000000000000000000000000000000 P 360bd4704b1148ec8fb3f750af7e33aa [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 22:51:04.795535 24204 raft_consensus.cc:482] T 00000000000000000000000000000000 P 360bd4704b1148ec8fb3f750af7e33aa [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 22:51:04.795583 24204 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 360bd4704b1148ec8fb3f750af7e33aa [term 0 FOLLOWER]: Advancing to term 1
I20260929 22:51:04.796156 24204 raft_consensus.cc:504] T 00000000000000000000000000000000 P 360bd4704b1148ec8fb3f750af7e33aa [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "360bd4704b1148ec8fb3f750af7e33aa" member_type: VOTER last_known_addr { host: "127.14.147.254" port: 46397 } }
I20260929 22:51:04.796257 24204 leader_election.cc:302] T 00000000000000000000000000000000 P 360bd4704b1148ec8fb3f750af7e33aa [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 360bd4704b1148ec8fb3f750af7e33aa; no voters:
I20260929 22:51:04.796448 24204 leader_election.cc:288] T 00000000000000000000000000000000 P 360bd4704b1148ec8fb3f750af7e33aa [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 22:51:04.796495 24209 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 360bd4704b1148ec8fb3f750af7e33aa [term 1 FOLLOWER]: Leader election won for term 1
I20260929 22:51:04.796665 24209 raft_consensus.cc:686] T 00000000000000000000000000000000 P 360bd4704b1148ec8fb3f750af7e33aa [term 1 LEADER]: Becoming Leader. State: Replica: 360bd4704b1148ec8fb3f750af7e33aa, State: Running, Role: LEADER
I20260929 22:51:04.796695 24204 sys_catalog.cc:564] T 00000000000000000000000000000000 P 360bd4704b1148ec8fb3f750af7e33aa [sys.catalog]: configured and running, proceeding with master startup.
I20260929 22:51:04.796828 24209 consensus_queue.cc:237] T 00000000000000000000000000000000 P 360bd4704b1148ec8fb3f750af7e33aa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "360bd4704b1148ec8fb3f750af7e33aa" member_type: VOTER last_known_addr { host: "127.14.147.254" port: 46397 } }
I20260929 22:51:04.797196 24210 sys_catalog.cc:455] T 00000000000000000000000000000000 P 360bd4704b1148ec8fb3f750af7e33aa [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "360bd4704b1148ec8fb3f750af7e33aa" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "360bd4704b1148ec8fb3f750af7e33aa" member_type: VOTER last_known_addr { host: "127.14.147.254" port: 46397 } } }
I20260929 22:51:04.797178 24211 sys_catalog.cc:455] T 00000000000000000000000000000000 P 360bd4704b1148ec8fb3f750af7e33aa [sys.catalog]: SysCatalogTable state changed. Reason: New leader 360bd4704b1148ec8fb3f750af7e33aa. Latest consensus state: current_term: 1 leader_uuid: "360bd4704b1148ec8fb3f750af7e33aa" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "360bd4704b1148ec8fb3f750af7e33aa" member_type: VOTER last_known_addr { host: "127.14.147.254" port: 46397 } } }
I20260929 22:51:04.797276 24210 sys_catalog.cc:458] T 00000000000000000000000000000000 P 360bd4704b1148ec8fb3f750af7e33aa [sys.catalog]: This master's current role is: LEADER
I20260929 22:51:04.797288 24211 sys_catalog.cc:458] T 00000000000000000000000000000000 P 360bd4704b1148ec8fb3f750af7e33aa [sys.catalog]: This master's current role is: LEADER
I20260929 22:51:04.797465 24217 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 22:51:04.797945 24217 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 22:51:04.799707 24217 catalog_manager.cc:1347] Generated new cluster ID: ded34ae8282d48df89df360f7bbc059d
I20260929 22:51:04.799757 24217 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 22:51:04.812672 24217 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 22:51:04.813207 24217 catalog_manager.cc:1504] Loading token signing keys...
I20260929 22:51:04.816753 24217 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 360bd4704b1148ec8fb3f750af7e33aa: Generated new TSK 0
I20260929 22:51:04.816933 24217 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 22:51:04.821026 14927 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhWOCLk/build/release/bin/kudu
/tmp/dist-test-taskhWOCLk/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790722213368738-14927-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790722213368738-14927-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790722213368738-14927-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790722213368738-14927-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.147.193:0
--local_ip_for_outbound_sockets=127.14.147.193
--webserver_interface=127.14.147.193
--webserver_port=0
--tserver_master_addrs=127.14.147.254:46397
--builtin_ntp_servers=127.14.147.212:33649
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20260929 22:51:04.902781 24228 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 22:51:04.903007 24228 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 22:51:04.903038 24228 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 22:51:04.904337 24228 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 22:51:04.904383 24228 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 22:51:04.904419 24228 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.147.193
I20260929 22:51:04.905879 24228 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.147.212:33649
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790722213368738-14927-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790722213368738-14927-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.147.193:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790722213368738-14927-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.14.147.193
--webserver_port=0
--tserver_master_addrs=127.14.147.254:46397
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24228
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.147.193
--log_dir=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790722213368738-14927-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 21:58:54 UTC on ccycloud.quasar-xguoti.root.comops.site
build id 4681390
I20260929 22:51:04.906138 24228 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 22:51:04.906456 24228 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 22:51:04.911496 24233 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 22:51:04.911597 24234 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 22:51:04.911891 24236 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 22:51:04.911846 24228 server_base.cc:1034] running on GCE node
I20260929 22:51:04.912156 24228 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 22:51:04.912492 24228 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 22:51:04.913692 24228 hybrid_clock.cc:648] HybridClock initialized: now 1790722264913661 us; error 47 us; skew 500 ppm
I20260929 22:51:04.915102 24228 webserver.cc:460] Webserver started at http://127.14.147.193:35023/ using document root <none> and password file <none>
I20260929 22:51:04.915342 24228 fs_manager.cc:359] Metadata directory not provided
I20260929 22:51:04.915409 24228 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 22:51:04.915572 24228 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 22:51:04.916666 24228 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790722213368738-14927-0/minicluster-data/ts-0/data/instance:
uuid: "4d49115109a648fb9302c03af620420e"
format_stamp: "Formatted at 2026-09-29 22:51:04 on dist-test-slave-98mk"
server_key: "3639b1d8e2213b8eda62fca0d90dc75c"
server_key_iv: "de038bf130fb98341439b7a12b33ab17"
server_key_version: "encryptionkey@0"
I20260929 22:51:04.917004 24228 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790722213368738-14927-0/minicluster-data/ts-0/wal/instance:
uuid: "4d49115109a648fb9302c03af620420e"
format_stamp: "Formatted at 2026-09-29 22:51:04 on dist-test-slave-98mk"
server_key: "3639b1d8e2213b8eda62fca0d90dc75c"
server_key_iv: "de038bf130fb98341439b7a12b33ab17"
server_key_version: "encryptionkey@0"
I20260929 22:51:04.918300 24228 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260929 22:51:04.919046 24242 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 22:51:04.919194 24228 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260929 22:51:04.919282 24228 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790722213368738-14927-0/minicluster-data/ts-0/data,/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790722213368738-14927-0/minicluster-data/ts-0/wal
uuid: "4d49115109a648fb9302c03af620420e"
format_stamp: "Formatted at 2026-09-29 22:51:04 on dist-test-slave-98mk"
server_key: "3639b1d8e2213b8eda62fca0d90dc75c"
server_key_iv: "de038bf130fb98341439b7a12b33ab17"
server_key_version: "encryptionkey@0"
I20260929 22:51:04.919366 24228 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790722213368738-14927-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790722213368738-14927-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790722213368738-14927-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 22:51:04.930307 24228 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 22:51:04.930636 24228 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 22:51:04.930765 24228 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 22:51:04.930987 24228 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 22:51:04.931304 24228 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 22:51:04.931344 24228 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 22:51:04.931368 24228 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 22:51:04.931391 24228 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 22:51:04.937155 24228 rpc_server.cc:307] RPC server started. Bound to: 127.14.147.193:33773
I20260929 22:51:04.937204 24356 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.147.193:33773 every 8 connection(s)
I20260929 22:51:04.937543 24228 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790722213368738-14927-0/minicluster-data/ts-0/data/info.pb
I20260929 22:51:04.942971 24207 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:51:04.938281 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.147.193:45257 (local address 127.14.147.254:46397)
0929 22:51:04.938338 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0929 22:51:04.938370 (+ 32us) server_negotiation.cc:399] Waiting for connection header
0929 22:51:04.938634 (+ 264us) server_negotiation.cc:407] Connection header received
0929 22:51:04.939132 (+ 498us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 22:51:04.939134 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 22:51:04.939147 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 22:51:04.939231 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0929 22:51:04.939934 (+ 703us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:51:04.940446 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:51:04.941488 (+ 1042us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:51:04.941698 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:51:04.941756 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 22:51:04.942704 (+ 948us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 22:51:04.942706 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 22:51:04.942706 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 22:51:04.942711 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0929 22:51:04.942726 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 22:51:04.942730 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 22:51:04.942781 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0929 22:51:04.942872 (+ 91us) server_negotiation.cc:299] Negotiation successful
0929 22:51:04.942904 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260929 22:51:04.943141 24359 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:51:04.938262 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.147.254:46397 (local address 127.14.147.193:45257)
0929 22:51:04.938452 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0929 22:51:04.938496 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0929 22:51:04.938986 (+ 490us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 22:51:04.939289 (+ 303us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 22:51:04.939293 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 22:51:04.939301 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0929 22:51:04.939816 (+ 515us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 22:51:04.939823 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 22:51:04.940566 (+ 743us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 22:51:04.940567 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 22:51:04.941390 (+ 823us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 22:51:04.941392 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 22:51:04.941800 (+ 408us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 22:51:04.941802 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 22:51:04.941929 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 22:51:04.942506 (+ 577us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 22:51:04.942507 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0929 22:51:04.942536 (+ 29us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 22:51:04.942561 (+ 25us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 22:51:04.942791 (+ 230us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 22:51:04.942794 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 22:51:04.942797 (+ 3us) client_negotiation.cc:769] Sending connection context
0929 22:51:04.942846 (+ 49us) client_negotiation.cc:240] Negotiation successful
0929 22:51:04.942937 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":90,"threads_started":1}
I20260929 22:51:04.943622 24357 heartbeater.cc:347] Connected to a master server at 127.14.147.254:46397
I20260929 22:51:04.943714 24357 heartbeater.cc:464] Registering TS with master...
I20260929 22:51:04.943898 24357 heartbeater.cc:511] Master 127.14.147.254:46397 requested a full tablet report, sending...
I20260929 22:51:04.944367 24168 ts_manager.cc:194] Registered new tserver with Master: 4d49115109a648fb9302c03af620420e (127.14.147.193:33773)
I20260929 22:51:04.945086 24168 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.14.147.193:45257
I20260929 22:51:04.945955 14927 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhWOCLk/build/release/bin/kudu as pid 24228
I20260929 22:51:04.946029 14927 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790722213368738-14927-0/minicluster-data/ts-0/wal/instance
I20260929 22:51:04.946182 14927 external_mini_cluster.cc:1428] Setting key 1c139bf2c80b11a4f048d68af327ed76
I20260929 22:51:04.946944 14927 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhWOCLk/build/release/bin/kudu
/tmp/dist-test-taskhWOCLk/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790722213368738-14927-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790722213368738-14927-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790722213368738-14927-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790722213368738-14927-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.147.194:0
--local_ip_for_outbound_sockets=127.14.147.194
--webserver_interface=127.14.147.194
--webserver_port=0
--tserver_master_addrs=127.14.147.254:46397
--builtin_ntp_servers=127.14.147.212:33649
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20260929 22:51:05.030050 24360 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 22:51:05.030300 24360 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 22:51:05.030330 24360 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 22:51:05.031661 24360 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 22:51:05.031704 24360 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 22:51:05.031731 24360 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.147.194
I20260929 22:51:05.033113 24360 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.147.212:33649
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790722213368738-14927-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790722213368738-14927-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.147.194:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790722213368738-14927-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.14.147.194
--webserver_port=0
--tserver_master_addrs=127.14.147.254:46397
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24360
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.147.194
--log_dir=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790722213368738-14927-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 21:58:54 UTC on ccycloud.quasar-xguoti.root.comops.site
build id 4681390
I20260929 22:51:05.033326 24360 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 22:51:05.033586 24360 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 22:51:05.038582 24366 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 22:51:05.038770 24360 server_base.cc:1034] running on GCE node
W20260929 22:51:05.038672 24365 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 22:51:05.038582 24368 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 22:51:05.039191 24360 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 22:51:05.039572 24360 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 22:51:05.040773 24360 hybrid_clock.cc:648] HybridClock initialized: now 1790722265040751 us; error 39 us; skew 500 ppm
I20260929 22:51:05.042269 24360 webserver.cc:460] Webserver started at http://127.14.147.194:41149/ using document root <none> and password file <none>
I20260929 22:51:05.042500 24360 fs_manager.cc:359] Metadata directory not provided
I20260929 22:51:05.042552 24360 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 22:51:05.042688 24360 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 22:51:05.043890 24360 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790722213368738-14927-0/minicluster-data/ts-1/data/instance:
uuid: "632121fcb9a4423589245e53141dad56"
format_stamp: "Formatted at 2026-09-29 22:51:05 on dist-test-slave-98mk"
server_key: "e65669f51560da627b6791c6c54e8fe2"
server_key_iv: "b06d0f3afbc2cfe5efb39d6ce03eab8d"
server_key_version: "encryptionkey@0"
I20260929 22:51:05.044224 24360 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790722213368738-14927-0/minicluster-data/ts-1/wal/instance:
uuid: "632121fcb9a4423589245e53141dad56"
format_stamp: "Formatted at 2026-09-29 22:51:05 on dist-test-slave-98mk"
server_key: "e65669f51560da627b6791c6c54e8fe2"
server_key_iv: "b06d0f3afbc2cfe5efb39d6ce03eab8d"
server_key_version: "encryptionkey@0"
I20260929 22:51:05.045693 24360 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.001s
I20260929 22:51:05.046756 24374 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 22:51:05.047013 24360 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260929 22:51:05.047114 24360 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790722213368738-14927-0/minicluster-data/ts-1/data,/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790722213368738-14927-0/minicluster-data/ts-1/wal
uuid: "632121fcb9a4423589245e53141dad56"
format_stamp: "Formatted at 2026-09-29 22:51:05 on dist-test-slave-98mk"
server_key: "e65669f51560da627b6791c6c54e8fe2"
server_key_iv: "b06d0f3afbc2cfe5efb39d6ce03eab8d"
server_key_version: "encryptionkey@0"
I20260929 22:51:05.047204 24360 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790722213368738-14927-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790722213368738-14927-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790722213368738-14927-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 22:51:05.055984 24360 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 22:51:05.056279 24360 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 22:51:05.056411 24360 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 22:51:05.056628 24360 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 22:51:05.056977 24360 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 22:51:05.057010 24360 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 22:51:05.057039 24360 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 22:51:05.057054 24360 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 22:51:05.062975 24360 rpc_server.cc:307] RPC server started. Bound to: 127.14.147.194:41181
I20260929 22:51:05.063030 24488 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.147.194:41181 every 8 connection(s)
I20260929 22:51:05.063359 24360 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790722213368738-14927-0/minicluster-data/ts-1/data/info.pb
I20260929 22:51:05.068570 24207 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:51:05.064098 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.147.194:57003 (local address 127.14.147.254:46397)
0929 22:51:05.064150 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0929 22:51:05.064153 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 22:51:05.064360 (+ 207us) server_negotiation.cc:407] Connection header received
0929 22:51:05.064898 (+ 538us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 22:51:05.064899 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 22:51:05.064912 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 22:51:05.064959 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0929 22:51:05.065657 (+ 698us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:51:05.066154 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:51:05.067150 (+ 996us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:51:05.067345 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:51:05.067411 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 22:51:05.068284 (+ 873us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 22:51:05.068285 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 22:51:05.068286 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 22:51:05.068291 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0929 22:51:05.068306 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 22:51:05.068309 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 22:51:05.068353 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0929 22:51:05.068441 (+ 88us) server_negotiation.cc:299] Negotiation successful
0929 22:51:05.068482 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260929 22:51:05.068768 24491 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:51:05.064065 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.147.254:46397 (local address 127.14.147.194:57003)
0929 22:51:05.064245 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0929 22:51:05.064280 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0929 22:51:05.064796 (+ 516us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 22:51:05.065046 (+ 250us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 22:51:05.065050 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 22:51:05.065061 (+ 11us) client_negotiation.cc:189] Negotiated authn=SASL
0929 22:51:05.065563 (+ 502us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 22:51:05.065567 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 22:51:05.066250 (+ 683us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 22:51:05.066251 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 22:51:05.067065 (+ 814us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 22:51:05.067067 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 22:51:05.067449 (+ 382us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 22:51:05.067450 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 22:51:05.067598 (+ 148us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 22:51:05.068136 (+ 538us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 22:51:05.068136 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0929 22:51:05.068167 (+ 31us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 22:51:05.068192 (+ 25us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 22:51:05.068361 (+ 169us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 22:51:05.068366 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 22:51:05.068370 (+ 4us) client_negotiation.cc:769] Sending connection context
0929 22:51:05.068436 (+ 66us) client_negotiation.cc:240] Negotiation successful
0929 22:51:05.068547 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":75,"threads_started":1}
I20260929 22:51:05.069317 24489 heartbeater.cc:347] Connected to a master server at 127.14.147.254:46397
I20260929 22:51:05.069401 24489 heartbeater.cc:464] Registering TS with master...
I20260929 22:51:05.069581 24489 heartbeater.cc:511] Master 127.14.147.254:46397 requested a full tablet report, sending...
I20260929 22:51:05.069916 24168 ts_manager.cc:194] Registered new tserver with Master: 632121fcb9a4423589245e53141dad56 (127.14.147.194:41181)
I20260929 22:51:05.070315 24168 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.14.147.194:57003
I20260929 22:51:05.071626 14927 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhWOCLk/build/release/bin/kudu as pid 24360
I20260929 22:51:05.071712 14927 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790722213368738-14927-0/minicluster-data/ts-1/wal/instance
I20260929 22:51:05.071897 14927 external_mini_cluster.cc:1428] Setting key cc7c43df3f4af048514dbbecef64a5c8
I20260929 22:51:05.073020 14927 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhWOCLk/build/release/bin/kudu
/tmp/dist-test-taskhWOCLk/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790722213368738-14927-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790722213368738-14927-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790722213368738-14927-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790722213368738-14927-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.147.195:0
--local_ip_for_outbound_sockets=127.14.147.195
--webserver_interface=127.14.147.195
--webserver_port=0
--tserver_master_addrs=127.14.147.254:46397
--builtin_ntp_servers=127.14.147.212:33649
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20260929 22:51:05.154037 24492 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 22:51:05.154261 24492 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 22:51:05.154285 24492 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 22:51:05.155586 24492 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 22:51:05.155637 24492 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 22:51:05.155664 24492 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.147.195
I20260929 22:51:05.157075 24492 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.147.212:33649
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790722213368738-14927-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790722213368738-14927-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.147.195:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790722213368738-14927-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.14.147.195
--webserver_port=0
--tserver_master_addrs=127.14.147.254:46397
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24492
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.147.195
--log_dir=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790722213368738-14927-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 21:58:54 UTC on ccycloud.quasar-xguoti.root.comops.site
build id 4681390
I20260929 22:51:05.157320 24492 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 22:51:05.157600 24492 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 22:51:05.162731 24498 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 22:51:05.162758 24500 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 22:51:05.162818 24492 server_base.cc:1034] running on GCE node
W20260929 22:51:05.162758 24497 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 22:51:05.163383 24492 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 22:51:05.163802 24492 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 22:51:05.165004 24492 hybrid_clock.cc:648] HybridClock initialized: now 1790722265164983 us; error 37 us; skew 500 ppm
I20260929 22:51:05.166395 24492 webserver.cc:460] Webserver started at http://127.14.147.195:46801/ using document root <none> and password file <none>
I20260929 22:51:05.166625 24492 fs_manager.cc:359] Metadata directory not provided
I20260929 22:51:05.166673 24492 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 22:51:05.166800 24492 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 22:51:05.167959 24492 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790722213368738-14927-0/minicluster-data/ts-2/data/instance:
uuid: "c3b39bbb20214d2eb88dd54badd02960"
format_stamp: "Formatted at 2026-09-29 22:51:05 on dist-test-slave-98mk"
server_key: "e59ae6d026ab3c89f75263e9fe9ad959"
server_key_iv: "5d8ed03de23f59a2a20d264f2bc335c4"
server_key_version: "encryptionkey@0"
I20260929 22:51:05.168298 24492 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790722213368738-14927-0/minicluster-data/ts-2/wal/instance:
uuid: "c3b39bbb20214d2eb88dd54badd02960"
format_stamp: "Formatted at 2026-09-29 22:51:05 on dist-test-slave-98mk"
server_key: "e59ae6d026ab3c89f75263e9fe9ad959"
server_key_iv: "5d8ed03de23f59a2a20d264f2bc335c4"
server_key_version: "encryptionkey@0"
I20260929 22:51:05.169581 24492 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260929 22:51:05.170373 24506 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 22:51:05.170552 24492 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260929 22:51:05.170640 24492 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790722213368738-14927-0/minicluster-data/ts-2/data,/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790722213368738-14927-0/minicluster-data/ts-2/wal
uuid: "c3b39bbb20214d2eb88dd54badd02960"
format_stamp: "Formatted at 2026-09-29 22:51:05 on dist-test-slave-98mk"
server_key: "e59ae6d026ab3c89f75263e9fe9ad959"
server_key_iv: "5d8ed03de23f59a2a20d264f2bc335c4"
server_key_version: "encryptionkey@0"
I20260929 22:51:05.170703 24492 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790722213368738-14927-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790722213368738-14927-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790722213368738-14927-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 22:51:05.182375 24492 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 22:51:05.182677 24492 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 22:51:05.182781 24492 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 22:51:05.182981 24492 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 22:51:05.183300 24492 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 22:51:05.183331 24492 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 22:51:05.183362 24492 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 22:51:05.183385 24492 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 22:51:05.188791 24492 rpc_server.cc:307] RPC server started. Bound to: 127.14.147.195:36167
I20260929 22:51:05.188865 24620 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.147.195:36167 every 8 connection(s)
I20260929 22:51:05.189152 24492 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790722213368738-14927-0/minicluster-data/ts-2/data/info.pb
I20260929 22:51:05.194026 24207 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:51:05.189838 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.147.195:33647 (local address 127.14.147.254:46397)
0929 22:51:05.189912 (+ 74us) server_negotiation.cc:206] Beginning negotiation
0929 22:51:05.189914 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 22:51:05.190147 (+ 233us) server_negotiation.cc:407] Connection header received
0929 22:51:05.190644 (+ 497us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 22:51:05.190645 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 22:51:05.190655 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 22:51:05.190706 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0929 22:51:05.191321 (+ 615us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:51:05.191821 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:51:05.192812 (+ 991us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:51:05.193007 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:51:05.193071 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 22:51:05.193809 (+ 738us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 22:51:05.193811 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 22:51:05.193811 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 22:51:05.193816 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0929 22:51:05.193830 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 22:51:05.193833 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 22:51:05.193872 (+ 39us) server_negotiation.cc:1035] Waiting for connection context
0929 22:51:05.193937 (+ 65us) server_negotiation.cc:299] Negotiation successful
0929 22:51:05.193961 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260929 22:51:05.194224 24623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:51:05.189819 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.147.254:46397 (local address 127.14.147.195:33647)
0929 22:51:05.190028 (+ 209us) negotiation.cc:107] Waiting for socket to connect
0929 22:51:05.190065 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0929 22:51:05.190562 (+ 497us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 22:51:05.190763 (+ 201us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 22:51:05.190767 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 22:51:05.190775 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0929 22:51:05.191256 (+ 481us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 22:51:05.191261 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 22:51:05.191897 (+ 636us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 22:51:05.191897 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 22:51:05.192697 (+ 800us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 22:51:05.192699 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 22:51:05.193079 (+ 380us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 22:51:05.193080 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 22:51:05.193190 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 22:51:05.193677 (+ 487us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 22:51:05.193677 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0929 22:51:05.193706 (+ 29us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 22:51:05.193730 (+ 24us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 22:51:05.193878 (+ 148us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 22:51:05.193881 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 22:51:05.193883 (+ 2us) client_negotiation.cc:769] Sending connection context
0929 22:51:05.193931 (+ 48us) client_negotiation.cc:240] Negotiation successful
0929 22:51:05.194022 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":146,"thread_start_us":99,"threads_started":1}
I20260929 22:51:05.194656 24621 heartbeater.cc:347] Connected to a master server at 127.14.147.254:46397
I20260929 22:51:05.194751 24621 heartbeater.cc:464] Registering TS with master...
I20260929 22:51:05.194941 24621 heartbeater.cc:511] Master 127.14.147.254:46397 requested a full tablet report, sending...
I20260929 22:51:05.195289 24168 ts_manager.cc:194] Registered new tserver with Master: c3b39bbb20214d2eb88dd54badd02960 (127.14.147.195:36167)
I20260929 22:51:05.195691 24168 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.14.147.195:33647
I20260929 22:51:05.197938 14927 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhWOCLk/build/release/bin/kudu as pid 24492
I20260929 22:51:05.198037 14927 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790722213368738-14927-0/minicluster-data/ts-2/wal/instance
I20260929 22:51:05.198222 14927 external_mini_cluster.cc:1428] Setting key cfb0ccfa0c8116a3dd7849c3d4b0f373
I20260929 22:51:05.199062 14927 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260929 22:51:05.200771 24207 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 22:51:05.200329 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39142 (local address 127.14.147.254:46397)
0929 22:51:05.200368 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0929 22:51:05.200370 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 22:51:05.200481 (+ 111us) server_negotiation.cc:407] Connection header received
0929 22:51:05.200534 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 22:51:05.200535 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 22:51:05.200543 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 22:51:05.200595 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0929 22:51:05.200708 (+ 113us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:39142: BlockingRecv error: recv got EOF from 127.0.0.1:39142 (error 108)
Metrics: {"server-negotiator.queue_time_us":18}
I20260929 22:51:05.200860 14927 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhWOCLk/build/release/bin/kudu with pid 24228
I20260929 22:51:05.205392 14927 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhWOCLk/build/release/bin/kudu with pid 24360
I20260929 22:51:05.209451 14927 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhWOCLk/build/release/bin/kudu with pid 24492
I20260929 22:51:05.213469 14927 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhWOCLk/build/release/bin/kudu with pid 24137
2026-09-29T22:51:05Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationInsecureCluster (584 ms)
[ RUN ] SecurityITest.TestRequireEncryptionInsecureCluster
2026-09-29T22:51:05Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T22:51:05Z Disabled control of system clock
I20260929 22:51:05.238265 14927 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhWOCLk/build/release/bin/kudu
/tmp/dist-test-taskhWOCLk/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790722213368738-14927-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790722213368738-14927-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790722213368738-14927-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790722213368738-14927-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.147.254:39025
--webserver_interface=127.14.147.254
--webserver_port=0
--builtin_ntp_servers=127.14.147.212:35631
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.147.254:39025
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260929 22:51:05.318969 24632 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 22:51:05.319192 24632 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 22:51:05.319214 24632 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 22:51:05.320411 24632 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 22:51:05.320461 24632 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 22:51:05.320472 24632 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 22:51:05.320483 24632 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 22:51:05.320494 24632 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 22:51:05.321852 24632 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.147.212:35631
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790722213368738-14927-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790722213368738-14927-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.14.147.254:39025
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.147.254:39025
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790722213368738-14927-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.14.147.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24632
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790722213368738-14927-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 21:58:54 UTC on ccycloud.quasar-xguoti.root.comops.site
build id 4681390
I20260929 22:51:05.322093 24632 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 22:51:05.322381 24632 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 22:51:05.327091 24638 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 22:51:05.327081 24640 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 22:51:05.327098 24637 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 22:51:05.327242 24632 server_base.cc:1034] running on GCE node
I20260929 22:51:05.327641 24632 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 22:51:05.327953 24632 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 22:51:05.329124 24632 hybrid_clock.cc:648] HybridClock initialized: now 1790722265329106 us; error 36 us; skew 500 ppm
I20260929 22:51:05.330407 24632 webserver.cc:460] Webserver started at http://127.14.147.254:45607/ using document root <none> and password file <none>
I20260929 22:51:05.330655 24632 fs_manager.cc:359] Metadata directory not provided
I20260929 22:51:05.330704 24632 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 22:51:05.330845 24632 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 22:51:05.331993 24632 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790722213368738-14927-0/minicluster-data/master-0/data/instance:
uuid: "b9894dd6afc1412f8d5bfeaeb4b4355a"
format_stamp: "Formatted at 2026-09-29 22:51:05 on dist-test-slave-98mk"
server_key: "b4256c9ea6986a8aebf104db65327eb7"
server_key_iv: "84b783cebee2a28d3ab8482ea52d3fce"
server_key_version: "encryptionkey@0"
I20260929 22:51:05.332329 24632 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790722213368738-14927-0/minicluster-data/master-0/wal/instance:
uuid: "b9894dd6afc1412f8d5bfeaeb4b4355a"
format_stamp: "Formatted at 2026-09-29 22:51:05 on dist-test-slave-98mk"
server_key: "b4256c9ea6986a8aebf104db65327eb7"
server_key_iv: "84b783cebee2a28d3ab8482ea52d3fce"
server_key_version: "encryptionkey@0"
I20260929 22:51:05.333765 24632 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260929 22:51:05.334604 24646 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 22:51:05.334791 24632 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260929 22:51:05.334874 24632 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790722213368738-14927-0/minicluster-data/master-0/data,/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790722213368738-14927-0/minicluster-data/master-0/wal
uuid: "b9894dd6afc1412f8d5bfeaeb4b4355a"
format_stamp: "Formatted at 2026-09-29 22:51:05 on dist-test-slave-98mk"
server_key: "b4256c9ea6986a8aebf104db65327eb7"
server_key_iv: "84b783cebee2a28d3ab8482ea52d3fce"
server_key_version: "encryptionkey@0"
I20260929 22:51:05.334944 24632 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790722213368738-14927-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790722213368738-14927-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790722213368738-14927-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 22:51:05.338358 24632 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 22:51:05.338634 24632 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 22:51:05.338757 24632 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 22:51:05.342383 24632 rpc_server.cc:307] RPC server started. Bound to: 127.14.147.254:39025
I20260929 22:51:05.342692 24699 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.147.254:39025 every 8 connection(s)
I20260929 22:51:05.342763 24632 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790722213368738-14927-0/minicluster-data/master-0/data/info.pb
I20260929 22:51:05.343475 24700 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 22:51:05.346314 24700 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b9894dd6afc1412f8d5bfeaeb4b4355a: Bootstrap starting.
I20260929 22:51:05.346906 24700 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b9894dd6afc1412f8d5bfeaeb4b4355a: Neither blocks nor log segments found. Creating new log.
I20260929 22:51:05.347205 24700 log.cc:824] T 00000000000000000000000000000000 P b9894dd6afc1412f8d5bfeaeb4b4355a: Log is configured to *not* fsync() on all Append() calls
I20260929 22:51:05.347949 24700 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b9894dd6afc1412f8d5bfeaeb4b4355a: No bootstrap required, opened a new log
I20260929 22:51:05.349586 24700 raft_consensus.cc:348] T 00000000000000000000000000000000 P b9894dd6afc1412f8d5bfeaeb4b4355a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b9894dd6afc1412f8d5bfeaeb4b4355a" member_type: VOTER last_known_addr { host: "127.14.147.254" port: 39025 } }
I20260929 22:51:05.349737 24700 raft_consensus.cc:374] T 00000000000000000000000000000000 P b9894dd6afc1412f8d5bfeaeb4b4355a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 22:51:05.349771 24700 raft_consensus.cc:729] T 00000000000000000000000000000000 P b9894dd6afc1412f8d5bfeaeb4b4355a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b9894dd6afc1412f8d5bfeaeb4b4355a, State: Initialized, Role: FOLLOWER
I20260929 22:51:05.349947 24700 consensus_queue.cc:260] T 00000000000000000000000000000000 P b9894dd6afc1412f8d5bfeaeb4b4355a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b9894dd6afc1412f8d5bfeaeb4b4355a" member_type: VOTER last_known_addr { host: "127.14.147.254" port: 39025 } }
I20260929 22:51:05.350013 24700 raft_consensus.cc:388] T 00000000000000000000000000000000 P b9894dd6afc1412f8d5bfeaeb4b4355a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 22:51:05.350036 24700 raft_consensus.cc:482] T 00000000000000000000000000000000 P b9894dd6afc1412f8d5bfeaeb4b4355a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 22:51:05.350082 24700 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b9894dd6afc1412f8d5bfeaeb4b4355a [term 0 FOLLOWER]: Advancing to term 1
I20260929 22:51:05.350668 24700 raft_consensus.cc:504] T 00000000000000000000000000000000 P b9894dd6afc1412f8d5bfeaeb4b4355a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b9894dd6afc1412f8d5bfeaeb4b4355a" member_type: VOTER last_known_addr { host: "127.14.147.254" port: 39025 } }
I20260929 22:51:05.350761 24700 leader_election.cc:302] T 00000000000000000000000000000000 P b9894dd6afc1412f8d5bfeaeb4b4355a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b9894dd6afc1412f8d5bfeaeb4b4355a; no voters:
I20260929 22:51:05.350941 24700 leader_election.cc:288] T 00000000000000000000000000000000 P b9894dd6afc1412f8d5bfeaeb4b4355a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 22:51:05.350981 24703 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b9894dd6afc1412f8d5bfeaeb4b4355a [term 1 FOLLOWER]: Leader election won for term 1
I20260929 22:51:05.351151 24703 raft_consensus.cc:686] T 00000000000000000000000000000000 P b9894dd6afc1412f8d5bfeaeb4b4355a [term 1 LEADER]: Becoming Leader. State: Replica: b9894dd6afc1412f8d5bfeaeb4b4355a, State: Running, Role: LEADER
I20260929 22:51:05.351188 24700 sys_catalog.cc:564] T 00000000000000000000000000000000 P b9894dd6afc1412f8d5bfeaeb4b4355a [sys.catalog]: configured and running, proceeding with master startup.
I20260929 22:51:05.351245 24703 consensus_queue.cc:237] T 00000000000000000000000000000000 P b9894dd6afc1412f8d5bfeaeb4b4355a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b9894dd6afc1412f8d5bfeaeb4b4355a" member_type: VOTER last_known_addr { host: "127.14.147.254" port: 39025 } }
I20260929 22:51:05.351655 24705 sys_catalog.cc:455] T 00000000000000000000000000000000 P b9894dd6afc1412f8d5bfeaeb4b4355a [sys.catalog]: SysCatalogTable state changed. Reason: New leader b9894dd6afc1412f8d5bfeaeb4b4355a. Latest consensus state: current_term: 1 leader_uuid: "b9894dd6afc1412f8d5bfeaeb4b4355a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b9894dd6afc1412f8d5bfeaeb4b4355a" member_type: VOTER last_known_addr { host: "127.14.147.254" port: 39025 } } }
I20260929 22:51:05.351744 24705 sys_catalog.cc:458] T 00000000000000000000000000000000 P b9894dd6afc1412f8d5bfeaeb4b4355a [sys.catalog]: This master's current role is: LEADER
I20260929 22:51:05.351861 24704 sys_catalog.cc:455] T 00000000000000000000000000000000 P b9894dd6afc1412f8d5bfeaeb4b4355a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b9894dd6afc1412f8d5bfeaeb4b4355a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b9894dd6afc1412f8d5bfeaeb4b4355a" member_type: VOTER last_known_addr { host: "127.14.147.254" port: 39025 } } }
I20260929 22:51:05.351922 24704 sys_catalog.cc:458] T 00000000000000000000000000000000 P b9894dd6afc1412f8d5bfeaeb4b4355a [sys.catalog]: This master's current role is: LEADER
I20260929 22:51:05.352157 24712 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 22:51:05.352643 24712 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 22:51:05.352829 14927 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhWOCLk/build/release/bin/kudu as pid 24632
I20260929 22:51:05.352921 14927 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790722213368738-14927-0/minicluster-data/master-0/wal/instance
I20260929 22:51:05.353108 14927 external_mini_cluster.cc:1428] Setting key 9e0f46b48cb240a0c1db2ef14f18549d
I20260929 22:51:05.354358 24712 catalog_manager.cc:1347] Generated new cluster ID: d34c96b7115945409dbb945cb32ff47a
I20260929 22:51:05.354415 24712 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 22:51:05.360822 24722 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:51:05.353566 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45134 (local address 127.14.147.254:39025)
0929 22:51:05.353743 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0929 22:51:05.353752 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 22:51:05.353789 (+ 37us) server_negotiation.cc:407] Connection header received
0929 22:51:05.353966 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 22:51:05.353973 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 22:51:05.354033 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 22:51:05.354182 (+ 149us) server_negotiation.cc:226] Negotiated authn=SASL
0929 22:51:05.360128 (+ 5946us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 22:51:05.360134 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 22:51:05.360134 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 22:51:05.360140 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0929 22:51:05.360204 (+ 64us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 22:51:05.360222 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 22:51:05.360303 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0929 22:51:05.360376 (+ 73us) server_negotiation.cc:299] Negotiation successful
0929 22:51:05.360574 (+ 198us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":104,"thread_start_us":78,"threads_started":1}
I20260929 22:51:05.376283 24712 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 22:51:05.376467 24712 catalog_manager.cc:1504] Loading token signing keys...
I20260929 22:51:05.386082 24712 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b9894dd6afc1412f8d5bfeaeb4b4355a: Generated new TSK 0
I20260929 22:51:05.386282 24712 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 22:51:05.391489 14927 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhWOCLk/build/release/bin/kudu
/tmp/dist-test-taskhWOCLk/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790722213368738-14927-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790722213368738-14927-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790722213368738-14927-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790722213368738-14927-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.147.193:0
--local_ip_for_outbound_sockets=127.14.147.193
--webserver_interface=127.14.147.193
--webserver_port=0
--tserver_master_addrs=127.14.147.254:39025
--builtin_ntp_servers=127.14.147.212:35631
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260929 22:51:05.476657 24724 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 22:51:05.476927 24724 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 22:51:05.476954 24724 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 22:51:05.479517 24724 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 22:51:05.479594 24724 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 22:51:05.479640 24724 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.147.193
I20260929 22:51:05.481449 24724 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.147.212:35631
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790722213368738-14927-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790722213368738-14927-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.147.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790722213368738-14927-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.14.147.193
--webserver_port=0
--tserver_master_addrs=127.14.147.254:39025
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24724
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.147.193
--log_dir=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790722213368738-14927-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 21:58:54 UTC on ccycloud.quasar-xguoti.root.comops.site
build id 4681390
I20260929 22:51:05.481741 24724 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 22:51:05.482066 24724 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 22:51:05.487017 24729 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 22:51:05.487043 24732 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 22:51:05.487175 24724 server_base.cc:1034] running on GCE node
W20260929 22:51:05.487035 24730 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 22:51:05.487522 24724 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 22:51:05.487782 24724 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 22:51:05.488951 24724 hybrid_clock.cc:648] HybridClock initialized: now 1790722265488938 us; error 29 us; skew 500 ppm
I20260929 22:51:05.490168 24724 webserver.cc:460] Webserver started at http://127.14.147.193:38729/ using document root <none> and password file <none>
I20260929 22:51:05.490427 24724 fs_manager.cc:359] Metadata directory not provided
I20260929 22:51:05.490478 24724 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 22:51:05.490612 24724 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 22:51:05.491727 24724 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790722213368738-14927-0/minicluster-data/ts-0/data/instance:
uuid: "1b01e66904934a38aa028eba06efdebd"
format_stamp: "Formatted at 2026-09-29 22:51:05 on dist-test-slave-98mk"
server_key: "b4b621e50a67bafea823a47b163dddaf"
server_key_iv: "482e16fa914f7c37b93a8d0fa9598ea3"
server_key_version: "encryptionkey@0"
I20260929 22:51:05.492066 24724 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790722213368738-14927-0/minicluster-data/ts-0/wal/instance:
uuid: "1b01e66904934a38aa028eba06efdebd"
format_stamp: "Formatted at 2026-09-29 22:51:05 on dist-test-slave-98mk"
server_key: "b4b621e50a67bafea823a47b163dddaf"
server_key_iv: "482e16fa914f7c37b93a8d0fa9598ea3"
server_key_version: "encryptionkey@0"
I20260929 22:51:05.493323 24724 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260929 22:51:05.494066 24738 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 22:51:05.494352 24724 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260929 22:51:05.494441 24724 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790722213368738-14927-0/minicluster-data/ts-0/data,/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790722213368738-14927-0/minicluster-data/ts-0/wal
uuid: "1b01e66904934a38aa028eba06efdebd"
format_stamp: "Formatted at 2026-09-29 22:51:05 on dist-test-slave-98mk"
server_key: "b4b621e50a67bafea823a47b163dddaf"
server_key_iv: "482e16fa914f7c37b93a8d0fa9598ea3"
server_key_version: "encryptionkey@0"
I20260929 22:51:05.494508 24724 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790722213368738-14927-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790722213368738-14927-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790722213368738-14927-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 22:51:05.497851 24724 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 22:51:05.498099 24724 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 22:51:05.498215 24724 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 22:51:05.498413 24724 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 22:51:05.498689 24724 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 22:51:05.498720 24724 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 22:51:05.498745 24724 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 22:51:05.498762 24724 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 22:51:05.504148 24724 rpc_server.cc:307] RPC server started. Bound to: 127.14.147.193:38603
I20260929 22:51:05.504205 24852 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.147.193:38603 every 8 connection(s)
I20260929 22:51:05.504511 24724 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790722213368738-14927-0/minicluster-data/ts-0/data/info.pb
I20260929 22:51:05.506563 14927 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhWOCLk/build/release/bin/kudu as pid 24724
I20260929 22:51:05.506673 14927 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790722213368738-14927-0/minicluster-data/ts-0/wal/instance
I20260929 22:51:05.506898 14927 external_mini_cluster.cc:1428] Setting key 9e9c0bcf204d90d482098e513c17f785
I20260929 22:51:05.507495 24722 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:51:05.505234 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.147.193:52703 (local address 127.14.147.254:39025)
0929 22:51:05.505299 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0929 22:51:05.505301 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 22:51:05.505476 (+ 175us) server_negotiation.cc:407] Connection header received
0929 22:51:05.505942 (+ 466us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 22:51:05.505944 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 22:51:05.505951 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 22:51:05.505991 (+ 40us) server_negotiation.cc:226] Negotiated authn=SASL
0929 22:51:05.507239 (+ 1248us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 22:51:05.507241 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 22:51:05.507241 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 22:51:05.507245 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0929 22:51:05.507264 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 22:51:05.507268 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 22:51:05.507320 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0929 22:51:05.507392 (+ 72us) server_negotiation.cc:299] Negotiation successful
0929 22:51:05.507410 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260929 22:51:05.507733 24855 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:51:05.505194 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.147.254:39025 (local address 127.14.147.193:52703)
0929 22:51:05.505365 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0929 22:51:05.505400 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0929 22:51:05.505852 (+ 452us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 22:51:05.506048 (+ 196us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 22:51:05.506051 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 22:51:05.506059 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0929 22:51:05.507074 (+ 1015us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 22:51:05.507075 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0929 22:51:05.507115 (+ 40us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 22:51:05.507148 (+ 33us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 22:51:05.507324 (+ 176us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 22:51:05.507329 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 22:51:05.507332 (+ 3us) client_negotiation.cc:769] Sending connection context
0929 22:51:05.507402 (+ 70us) client_negotiation.cc:240] Negotiation successful
0929 22:51:05.507497 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":74,"threads_started":1}
I20260929 22:51:05.508203 24853 heartbeater.cc:347] Connected to a master server at 127.14.147.254:39025
I20260929 22:51:05.508314 24853 heartbeater.cc:464] Registering TS with master...
I20260929 22:51:05.508472 24853 heartbeater.cc:511] Master 127.14.147.254:39025 requested a full tablet report, sending...
I20260929 22:51:05.509179 24663 ts_manager.cc:194] Registered new tserver with Master: 1b01e66904934a38aa028eba06efdebd (127.14.147.193:38603)
I20260929 22:51:05.511348 14927 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhWOCLk/build/release/bin/kudu
/tmp/dist-test-taskhWOCLk/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790722213368738-14927-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790722213368738-14927-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790722213368738-14927-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790722213368738-14927-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.147.194:0
--local_ip_for_outbound_sockets=127.14.147.194
--webserver_interface=127.14.147.194
--webserver_port=0
--tserver_master_addrs=127.14.147.254:39025
--builtin_ntp_servers=127.14.147.212:35631
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260929 22:51:05.592285 24856 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 22:51:05.592519 24856 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 22:51:05.592543 24856 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 22:51:05.593762 24856 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 22:51:05.593807 24856 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 22:51:05.593833 24856 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.147.194
I20260929 22:51:05.595237 24856 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.147.212:35631
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790722213368738-14927-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790722213368738-14927-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.147.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790722213368738-14927-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.14.147.194
--webserver_port=0
--tserver_master_addrs=127.14.147.254:39025
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24856
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.147.194
--log_dir=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790722213368738-14927-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 21:58:54 UTC on ccycloud.quasar-xguoti.root.comops.site
build id 4681390
I20260929 22:51:05.595554 24856 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 22:51:05.595851 24856 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 22:51:05.600675 24862 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 22:51:05.600694 24856 server_base.cc:1034] running on GCE node
W20260929 22:51:05.600694 24861 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 22:51:05.600653 24864 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 22:51:05.601048 24856 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 22:51:05.601302 24856 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 22:51:05.602465 24856 hybrid_clock.cc:648] HybridClock initialized: now 1790722265602450 us; error 28 us; skew 500 ppm
I20260929 22:51:05.603808 24856 webserver.cc:460] Webserver started at http://127.14.147.194:36999/ using document root <none> and password file <none>
I20260929 22:51:05.604019 24856 fs_manager.cc:359] Metadata directory not provided
I20260929 22:51:05.604067 24856 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 22:51:05.604192 24856 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 22:51:05.605235 24856 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790722213368738-14927-0/minicluster-data/ts-1/data/instance:
uuid: "bc4a549f6a0c4fd781699f51d746a301"
format_stamp: "Formatted at 2026-09-29 22:51:05 on dist-test-slave-98mk"
server_key: "981326c61e018657f55ee3387873c4c5"
server_key_iv: "79673e66a41d3e0f33115b9454604d1b"
server_key_version: "encryptionkey@0"
I20260929 22:51:05.605561 24856 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790722213368738-14927-0/minicluster-data/ts-1/wal/instance:
uuid: "bc4a549f6a0c4fd781699f51d746a301"
format_stamp: "Formatted at 2026-09-29 22:51:05 on dist-test-slave-98mk"
server_key: "981326c61e018657f55ee3387873c4c5"
server_key_iv: "79673e66a41d3e0f33115b9454604d1b"
server_key_version: "encryptionkey@0"
I20260929 22:51:05.606752 24856 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260929 22:51:05.607360 24870 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 22:51:05.607523 24856 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260929 22:51:05.607600 24856 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790722213368738-14927-0/minicluster-data/ts-1/data,/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790722213368738-14927-0/minicluster-data/ts-1/wal
uuid: "bc4a549f6a0c4fd781699f51d746a301"
format_stamp: "Formatted at 2026-09-29 22:51:05 on dist-test-slave-98mk"
server_key: "981326c61e018657f55ee3387873c4c5"
server_key_iv: "79673e66a41d3e0f33115b9454604d1b"
server_key_version: "encryptionkey@0"
I20260929 22:51:05.607666 24856 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790722213368738-14927-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790722213368738-14927-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790722213368738-14927-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 22:51:05.610831 24856 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 22:51:05.611091 24856 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 22:51:05.611209 24856 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 22:51:05.611394 24856 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 22:51:05.611768 24856 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 22:51:05.611800 24856 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 22:51:05.611826 24856 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 22:51:05.611851 24856 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 22:51:05.617228 24856 rpc_server.cc:307] RPC server started. Bound to: 127.14.147.194:40073
I20260929 22:51:05.617275 24983 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.147.194:40073 every 8 connection(s)
I20260929 22:51:05.617588 24856 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790722213368738-14927-0/minicluster-data/ts-1/data/info.pb
I20260929 22:51:05.620398 24722 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:51:05.618225 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.147.194:52325 (local address 127.14.147.254:39025)
0929 22:51:05.618312 (+ 87us) server_negotiation.cc:206] Beginning negotiation
0929 22:51:05.618315 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 22:51:05.618485 (+ 170us) server_negotiation.cc:407] Connection header received
0929 22:51:05.618957 (+ 472us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 22:51:05.618958 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 22:51:05.618964 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 22:51:05.619017 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0929 22:51:05.620178 (+ 1161us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 22:51:05.620179 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 22:51:05.620179 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 22:51:05.620183 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0929 22:51:05.620200 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 22:51:05.620204 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 22:51:05.620263 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0929 22:51:05.620326 (+ 63us) server_negotiation.cc:299] Negotiation successful
0929 22:51:05.620337 (+ 11us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260929 22:51:05.620571 24986 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:51:05.618202 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.147.254:39025 (local address 127.14.147.194:52325)
0929 22:51:05.618365 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0929 22:51:05.618399 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0929 22:51:05.618846 (+ 447us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 22:51:05.619069 (+ 223us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 22:51:05.619072 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 22:51:05.619079 (+ 7us) client_negotiation.cc:189] Negotiated authn=SASL
0929 22:51:05.620001 (+ 922us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 22:51:05.620002 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0929 22:51:05.620031 (+ 29us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 22:51:05.620060 (+ 29us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 22:51:05.620266 (+ 206us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 22:51:05.620269 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 22:51:05.620271 (+ 2us) client_negotiation.cc:769] Sending connection context
0929 22:51:05.620322 (+ 51us) client_negotiation.cc:240] Negotiation successful
0929 22:51:05.620371 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":83,"threads_started":1}
I20260929 22:51:05.621073 24984 heartbeater.cc:347] Connected to a master server at 127.14.147.254:39025
I20260929 22:51:05.621150 24984 heartbeater.cc:464] Registering TS with master...
I20260929 22:51:05.621263 24984 heartbeater.cc:511] Master 127.14.147.254:39025 requested a full tablet report, sending...
I20260929 22:51:05.621582 24663 ts_manager.cc:194] Registered new tserver with Master: bc4a549f6a0c4fd781699f51d746a301 (127.14.147.194:40073)
I20260929 22:51:05.626507 14927 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhWOCLk/build/release/bin/kudu as pid 24856
I20260929 22:51:05.626606 14927 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790722213368738-14927-0/minicluster-data/ts-1/wal/instance
I20260929 22:51:05.626794 14927 external_mini_cluster.cc:1428] Setting key b2390cec342bac7ddf74c9125259eeef
I20260929 22:51:05.627765 14927 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhWOCLk/build/release/bin/kudu
/tmp/dist-test-taskhWOCLk/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790722213368738-14927-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790722213368738-14927-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790722213368738-14927-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790722213368738-14927-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.147.195:0
--local_ip_for_outbound_sockets=127.14.147.195
--webserver_interface=127.14.147.195
--webserver_port=0
--tserver_master_addrs=127.14.147.254:39025
--builtin_ntp_servers=127.14.147.212:35631
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260929 22:51:05.708262 24987 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 22:51:05.708484 24987 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 22:51:05.708508 24987 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 22:51:05.709733 24987 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 22:51:05.709784 24987 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 22:51:05.709810 24987 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.147.195
I20260929 22:51:05.711313 24987 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.147.212:35631
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790722213368738-14927-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790722213368738-14927-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.147.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790722213368738-14927-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.14.147.195
--webserver_port=0
--tserver_master_addrs=127.14.147.254:39025
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24987
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.147.195
--log_dir=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790722213368738-14927-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 21:58:54 UTC on ccycloud.quasar-xguoti.root.comops.site
build id 4681390
I20260929 22:51:05.711560 24987 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 22:51:05.711830 24987 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 22:51:05.716593 24993 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 22:51:05.716619 24987 server_base.cc:1034] running on GCE node
W20260929 22:51:05.716588 24992 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 22:51:05.716586 24995 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 22:51:05.717001 24987 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 22:51:05.717226 24987 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 22:51:05.718374 24987 hybrid_clock.cc:648] HybridClock initialized: now 1790722265718358 us; error 28 us; skew 500 ppm
I20260929 22:51:05.719666 24987 webserver.cc:460] Webserver started at http://127.14.147.195:44415/ using document root <none> and password file <none>
I20260929 22:51:05.719928 24987 fs_manager.cc:359] Metadata directory not provided
I20260929 22:51:05.719990 24987 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 22:51:05.720094 24987 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 22:51:05.721176 24987 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790722213368738-14927-0/minicluster-data/ts-2/data/instance:
uuid: "b14354a6ec0b41b0a03bbfe3285298b3"
format_stamp: "Formatted at 2026-09-29 22:51:05 on dist-test-slave-98mk"
server_key: "02a5d7eedfbb9bca1504cd32e477722d"
server_key_iv: "db4be4c6fff0b9de153e05bf5ada8e7b"
server_key_version: "encryptionkey@0"
I20260929 22:51:05.721501 24987 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790722213368738-14927-0/minicluster-data/ts-2/wal/instance:
uuid: "b14354a6ec0b41b0a03bbfe3285298b3"
format_stamp: "Formatted at 2026-09-29 22:51:05 on dist-test-slave-98mk"
server_key: "02a5d7eedfbb9bca1504cd32e477722d"
server_key_iv: "db4be4c6fff0b9de153e05bf5ada8e7b"
server_key_version: "encryptionkey@0"
I20260929 22:51:05.722743 24987 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260929 22:51:05.723489 25002 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 22:51:05.723654 24987 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260929 22:51:05.723721 24987 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790722213368738-14927-0/minicluster-data/ts-2/data,/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790722213368738-14927-0/minicluster-data/ts-2/wal
uuid: "b14354a6ec0b41b0a03bbfe3285298b3"
format_stamp: "Formatted at 2026-09-29 22:51:05 on dist-test-slave-98mk"
server_key: "02a5d7eedfbb9bca1504cd32e477722d"
server_key_iv: "db4be4c6fff0b9de153e05bf5ada8e7b"
server_key_version: "encryptionkey@0"
I20260929 22:51:05.723790 24987 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790722213368738-14927-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790722213368738-14927-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790722213368738-14927-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 22:51:05.727087 24987 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 22:51:05.727341 24987 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 22:51:05.727487 24987 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 22:51:05.727705 24987 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 22:51:05.728021 24987 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 22:51:05.728051 24987 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 22:51:05.728080 24987 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 22:51:05.728101 24987 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 22:51:05.733752 24987 rpc_server.cc:307] RPC server started. Bound to: 127.14.147.195:39331
I20260929 22:51:05.733839 25115 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.147.195:39331 every 8 connection(s)
I20260929 22:51:05.734123 24987 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790722213368738-14927-0/minicluster-data/ts-2/data/info.pb
I20260929 22:51:05.737105 24722 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:51:05.734821 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.147.195:37599 (local address 127.14.147.254:39025)
0929 22:51:05.734885 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0929 22:51:05.734886 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0929 22:51:05.735096 (+ 210us) server_negotiation.cc:407] Connection header received
0929 22:51:05.735644 (+ 548us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 22:51:05.735645 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 22:51:05.735652 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 22:51:05.735704 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0929 22:51:05.736844 (+ 1140us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 22:51:05.736845 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 22:51:05.736846 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 22:51:05.736851 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0929 22:51:05.736867 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 22:51:05.736871 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 22:51:05.736930 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0929 22:51:05.737011 (+ 81us) server_negotiation.cc:299] Negotiation successful
0929 22:51:05.737028 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260929 22:51:05.737293 25118 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:51:05.734812 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.147.254:39025 (local address 127.14.147.195:37599)
0929 22:51:05.734983 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0929 22:51:05.735018 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0929 22:51:05.735543 (+ 525us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 22:51:05.735774 (+ 231us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 22:51:05.735778 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 22:51:05.735786 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0929 22:51:05.736687 (+ 901us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 22:51:05.736688 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0929 22:51:05.736719 (+ 31us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 22:51:05.736751 (+ 32us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 22:51:05.736944 (+ 193us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 22:51:05.736950 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 22:51:05.736953 (+ 3us) client_negotiation.cc:769] Sending connection context
0929 22:51:05.737016 (+ 63us) client_negotiation.cc:240] Negotiation successful
0929 22:51:05.737099 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":74,"threads_started":1}
I20260929 22:51:05.737738 25116 heartbeater.cc:347] Connected to a master server at 127.14.147.254:39025
I20260929 22:51:05.737816 25116 heartbeater.cc:464] Registering TS with master...
I20260929 22:51:05.737954 25116 heartbeater.cc:511] Master 127.14.147.254:39025 requested a full tablet report, sending...
I20260929 22:51:05.738293 24663 ts_manager.cc:194] Registered new tserver with Master: b14354a6ec0b41b0a03bbfe3285298b3 (127.14.147.195:39331)
I20260929 22:51:05.742539 14927 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhWOCLk/build/release/bin/kudu as pid 24987
I20260929 22:51:05.742628 14927 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790722213368738-14927-0/minicluster-data/ts-2/wal/instance
I20260929 22:51:05.742801 14927 external_mini_cluster.cc:1428] Setting key 288ffdc4f591b1e03f2ee718ce5d5807
I20260929 22:51:05.743667 14927 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260929 22:51:05.745405 24722 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 22:51:05.744935 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45146 (local address 127.14.147.254:39025)
0929 22:51:05.744995 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0929 22:51:05.744996 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0929 22:51:05.745111 (+ 115us) server_negotiation.cc:407] Connection header received
0929 22:51:05.745166 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 22:51:05.745167 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 22:51:05.745172 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 22:51:05.745218 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0929 22:51:05.745335 (+ 117us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:45146: BlockingRecv error: recv got EOF from 127.0.0.1:45146 (error 108)
Metrics: {"server-negotiator.queue_time_us":37}
I20260929 22:51:05.745441 14927 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhWOCLk/build/release/bin/kudu with pid 24724
I20260929 22:51:05.750316 14927 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhWOCLk/build/release/bin/kudu with pid 24856
I20260929 22:51:05.754130 14927 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhWOCLk/build/release/bin/kudu with pid 24987
I20260929 22:51:05.758392 14927 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhWOCLk/build/release/bin/kudu with pid 24632
2026-09-29T22:51:05Z chronyd exiting
[ OK ] SecurityITest.TestRequireEncryptionInsecureCluster (545 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationSecureCluster
Loading random data
Initializing database '/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790722213368738-14927-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 22:51:05 dist-test-slave-98mk krb5kdc[25126](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 22:51:05 dist-test-slave-98mk krb5kdc[25126](info): set up 2 sockets
Sep 29 22:51:05 dist-test-slave-98mk krb5kdc[25126](info): commencing operation
krb5kdc: starting...
W20260929 22:51:06.720827 14927 mini_kdc.cc:121] Time spent starting KDC: real 0.944s user 0.005s sys 0.000s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790722213368738-14927-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790722213368738-14927-0/krb5kdc/test-user.keytab.
Sep 29 22:51:06 dist-test-slave-98mk krb5kdc[25126](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790722266, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T22:51:06Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T22:51:06Z Disabled control of system clock
WARNING: no policy specified for kudu/127.14.147.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.147.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.147.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790722213368738-14927-0/krb5kdc/kudu_127.14.147.254.keytab.
Entry for principal kudu/127.14.147.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790722213368738-14927-0/krb5kdc/kudu_127.14.147.254.keytab.
I20260929 22:51:06.838093 14927 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhWOCLk/build/release/bin/kudu
/tmp/dist-test-taskhWOCLk/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790722213368738-14927-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790722213368738-14927-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790722213368738-14927-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790722213368738-14927-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.147.254:39875
--webserver_interface=127.14.147.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790722213368738-14927-0/krb5kdc/kudu_127.14.147.254.keytab
--principal=kudu/127.14.147.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.147.212:37831
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.147.254:39875
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790722213368738-14927-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790722213368738-14927-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790722213368738-14927-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 22:51:06.917976 25139 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 22:51:06.918210 25139 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 22:51:06.918232 25139 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 22:51:06.919507 25139 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 22:51:06.919556 25139 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 22:51:06.919570 25139 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 22:51:06.919581 25139 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 22:51:06.919591 25139 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 22:51:06.921147 25139 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.147.212:37831
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790722213368738-14927-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790722213368738-14927-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.14.147.254:39875
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790722213368738-14927-0/krb5kdc/kudu_127.14.147.254.keytab
--principal=kudu/127.14.147.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.147.254:39875
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790722213368738-14927-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.14.147.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25139
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790722213368738-14927-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 21:58:54 UTC on ccycloud.quasar-xguoti.root.comops.site
build id 4681390
I20260929 22:51:06.921387 25139 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 22:51:06.921604 25139 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 22:51:06.926265 25147 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 22:51:06.926251 25144 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 22:51:06.926251 25145 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 22:51:06.926283 25139 server_base.cc:1034] running on GCE node
I20260929 22:51:06.926668 25139 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 22:51:06.926937 25139 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 22:51:06.928082 25139 hybrid_clock.cc:648] HybridClock initialized: now 1790722266928069 us; error 29 us; skew 500 ppm
Sep 29 22:51:06 dist-test-slave-98mk krb5kdc[25126](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790722266, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.147.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 22:51:06.930394 25139 init.cc:377] Logged in from keytab as kudu/127.14.147.254@KRBTEST.COM (short username kudu)
I20260929 22:51:06.930801 25139 webserver.cc:460] Webserver started at http://127.14.147.254:35591/ using document root <none> and password file <none>
I20260929 22:51:06.931017 25139 fs_manager.cc:359] Metadata directory not provided
I20260929 22:51:06.931064 25139 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 22:51:06.931192 25139 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 22:51:06.932291 25139 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790722213368738-14927-0/minicluster-data/master-0/data/instance:
uuid: "51708af5c8894488ad0edde82284d09c"
format_stamp: "Formatted at 2026-09-29 22:51:06 on dist-test-slave-98mk"
server_key: "2a3b6d5c079d22c394170dedf35bc6ec"
server_key_iv: "4ccaf526a1b7b3ddb1d77c577b05e8c9"
server_key_version: "encryptionkey@0"
I20260929 22:51:06.932638 25139 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790722213368738-14927-0/minicluster-data/master-0/wal/instance:
uuid: "51708af5c8894488ad0edde82284d09c"
format_stamp: "Formatted at 2026-09-29 22:51:06 on dist-test-slave-98mk"
server_key: "2a3b6d5c079d22c394170dedf35bc6ec"
server_key_iv: "4ccaf526a1b7b3ddb1d77c577b05e8c9"
server_key_version: "encryptionkey@0"
I20260929 22:51:06.933954 25139 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260929 22:51:06.934636 25154 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 22:51:06.934782 25139 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260929 22:51:06.934859 25139 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790722213368738-14927-0/minicluster-data/master-0/data,/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790722213368738-14927-0/minicluster-data/master-0/wal
uuid: "51708af5c8894488ad0edde82284d09c"
format_stamp: "Formatted at 2026-09-29 22:51:06 on dist-test-slave-98mk"
server_key: "2a3b6d5c079d22c394170dedf35bc6ec"
server_key_iv: "4ccaf526a1b7b3ddb1d77c577b05e8c9"
server_key_version: "encryptionkey@0"
I20260929 22:51:06.934921 25139 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790722213368738-14927-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790722213368738-14927-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790722213368738-14927-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 22:51:06.950960 25139 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 22:51:06.953073 25139 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 22:51:06.953214 25139 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 22:51:06.956862 25139 rpc_server.cc:307] RPC server started. Bound to: 127.14.147.254:39875
I20260929 22:51:06.956908 25207 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.147.254:39875 every 8 connection(s)
I20260929 22:51:06.957229 25139 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790722213368738-14927-0/minicluster-data/master-0/data/info.pb
I20260929 22:51:06.958170 25208 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 22:51:06.960428 25208 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 51708af5c8894488ad0edde82284d09c: Bootstrap starting.
I20260929 22:51:06.960979 25208 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 51708af5c8894488ad0edde82284d09c: Neither blocks nor log segments found. Creating new log.
I20260929 22:51:06.961237 25208 log.cc:824] T 00000000000000000000000000000000 P 51708af5c8894488ad0edde82284d09c: Log is configured to *not* fsync() on all Append() calls
I20260929 22:51:06.961891 25208 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 51708af5c8894488ad0edde82284d09c: No bootstrap required, opened a new log
I20260929 22:51:06.963478 25208 raft_consensus.cc:348] T 00000000000000000000000000000000 P 51708af5c8894488ad0edde82284d09c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "51708af5c8894488ad0edde82284d09c" member_type: VOTER last_known_addr { host: "127.14.147.254" port: 39875 } }
I20260929 22:51:06.963620 25208 raft_consensus.cc:374] T 00000000000000000000000000000000 P 51708af5c8894488ad0edde82284d09c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 22:51:06.963641 25208 raft_consensus.cc:729] T 00000000000000000000000000000000 P 51708af5c8894488ad0edde82284d09c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51708af5c8894488ad0edde82284d09c, State: Initialized, Role: FOLLOWER
I20260929 22:51:06.963639 14927 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhWOCLk/build/release/bin/kudu as pid 25139
I20260929 22:51:06.963734 14927 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790722213368738-14927-0/minicluster-data/master-0/wal/instance
I20260929 22:51:06.963773 25208 consensus_queue.cc:260] T 00000000000000000000000000000000 P 51708af5c8894488ad0edde82284d09c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "51708af5c8894488ad0edde82284d09c" member_type: VOTER last_known_addr { host: "127.14.147.254" port: 39875 } }
I20260929 22:51:06.963845 25208 raft_consensus.cc:388] T 00000000000000000000000000000000 P 51708af5c8894488ad0edde82284d09c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 22:51:06.963881 25208 raft_consensus.cc:482] T 00000000000000000000000000000000 P 51708af5c8894488ad0edde82284d09c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 22:51:06.963937 25208 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 51708af5c8894488ad0edde82284d09c [term 0 FOLLOWER]: Advancing to term 1
I20260929 22:51:06.963943 14927 external_mini_cluster.cc:1428] Setting key 001147762db708e9be3d27c7d971ecc6
I20260929 22:51:06.965010 25208 raft_consensus.cc:504] T 00000000000000000000000000000000 P 51708af5c8894488ad0edde82284d09c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "51708af5c8894488ad0edde82284d09c" member_type: VOTER last_known_addr { host: "127.14.147.254" port: 39875 } }
I20260929 22:51:06.965121 25208 leader_election.cc:302] T 00000000000000000000000000000000 P 51708af5c8894488ad0edde82284d09c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 51708af5c8894488ad0edde82284d09c; no voters:
I20260929 22:51:06.965268 25208 leader_election.cc:288] T 00000000000000000000000000000000 P 51708af5c8894488ad0edde82284d09c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 22:51:06.965292 25213 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 51708af5c8894488ad0edde82284d09c [term 1 FOLLOWER]: Leader election won for term 1
I20260929 22:51:06.965389 25213 raft_consensus.cc:686] T 00000000000000000000000000000000 P 51708af5c8894488ad0edde82284d09c [term 1 LEADER]: Becoming Leader. State: Replica: 51708af5c8894488ad0edde82284d09c, State: Running, Role: LEADER
I20260929 22:51:06.965477 25213 consensus_queue.cc:237] T 00000000000000000000000000000000 P 51708af5c8894488ad0edde82284d09c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "51708af5c8894488ad0edde82284d09c" member_type: VOTER last_known_addr { host: "127.14.147.254" port: 39875 } }
I20260929 22:51:06.965520 25208 sys_catalog.cc:564] T 00000000000000000000000000000000 P 51708af5c8894488ad0edde82284d09c [sys.catalog]: configured and running, proceeding with master startup.
I20260929 22:51:06.965778 25213 sys_catalog.cc:455] T 00000000000000000000000000000000 P 51708af5c8894488ad0edde82284d09c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "51708af5c8894488ad0edde82284d09c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "51708af5c8894488ad0edde82284d09c" member_type: VOTER last_known_addr { host: "127.14.147.254" port: 39875 } } }
I20260929 22:51:06.965871 25213 sys_catalog.cc:458] T 00000000000000000000000000000000 P 51708af5c8894488ad0edde82284d09c [sys.catalog]: This master's current role is: LEADER
I20260929 22:51:06.965978 25213 sys_catalog.cc:455] T 00000000000000000000000000000000 P 51708af5c8894488ad0edde82284d09c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 51708af5c8894488ad0edde82284d09c. Latest consensus state: current_term: 1 leader_uuid: "51708af5c8894488ad0edde82284d09c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "51708af5c8894488ad0edde82284d09c" member_type: VOTER last_known_addr { host: "127.14.147.254" port: 39875 } } }
I20260929 22:51:06.966034 25213 sys_catalog.cc:458] T 00000000000000000000000000000000 P 51708af5c8894488ad0edde82284d09c [sys.catalog]: This master's current role is: LEADER
I20260929 22:51:06.966161 25218 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 22:51:06.966702 25218 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 22:51:06.968433 25218 catalog_manager.cc:1347] Generated new cluster ID: 6f6fe2c62fb3457facf3aa9b6e51cdfa
I20260929 22:51:06.968488 25218 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 29 22:51:06 dist-test-slave-98mk krb5kdc[25126](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790722266, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.147.254@KRBTEST.COM
I20260929 22:51:06.976142 25212 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:51:06.964430 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49486 (local address 127.14.147.254:39875)
0929 22:51:06.964641 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0929 22:51:06.964652 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0929 22:51:06.964692 (+ 40us) server_negotiation.cc:407] Connection header received
0929 22:51:06.964837 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 22:51:06.964845 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 22:51:06.964904 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 22:51:06.965052 (+ 148us) server_negotiation.cc:226] Negotiated authn=SASL
0929 22:51:06.965810 (+ 758us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:51:06.967093 (+ 1283us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:51:06.968152 (+ 1059us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:51:06.968742 (+ 590us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:51:06.968847 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 22:51:06.971661 (+ 2814us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 22:51:06.971665 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 22:51:06.971666 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 22:51:06.971682 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0929 22:51:06.974368 (+ 2686us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 22:51:06.974797 (+ 429us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 22:51:06.974800 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 22:51:06.974800 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 22:51:06.974949 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 22:51:06.975124 (+ 175us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 22:51:06.975124 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 22:51:06.975125 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 22:51:06.975500 (+ 375us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 22:51:06.975674 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0929 22:51:06.975747 (+ 73us) server_negotiation.cc:299] Negotiation successful
0929 22:51:06.975913 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":152,"thread_start_us":104,"threads_started":1}
I20260929 22:51:06.982450 25218 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 22:51:06.983075 25218 catalog_manager.cc:1504] Loading token signing keys...
I20260929 22:51:06.988013 25218 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 51708af5c8894488ad0edde82284d09c: Generated new TSK 0
I20260929 22:51:06.988238 25218 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.14.147.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.147.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.147.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790722213368738-14927-0/krb5kdc/kudu_127.14.147.193.keytab.
Entry for principal kudu/127.14.147.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790722213368738-14927-0/krb5kdc/kudu_127.14.147.193.keytab.
I20260929 22:51:07.014025 14927 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhWOCLk/build/release/bin/kudu
/tmp/dist-test-taskhWOCLk/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790722213368738-14927-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790722213368738-14927-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790722213368738-14927-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790722213368738-14927-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.147.193:0
--local_ip_for_outbound_sockets=127.14.147.193
--webserver_interface=127.14.147.193
--webserver_port=0
--tserver_master_addrs=127.14.147.254:39875
--keytab_file=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790722213368738-14927-0/krb5kdc/kudu_127.14.147.193.keytab
--principal=kudu/127.14.147.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.147.212:37831
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790722213368738-14927-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790722213368738-14927-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790722213368738-14927-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 22:51:07.097743 25234 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 22:51:07.097980 25234 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 22:51:07.098016 25234 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 22:51:07.099349 25234 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 22:51:07.099395 25234 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 22:51:07.099463 25234 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.147.193
I20260929 22:51:07.101063 25234 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.147.212:37831
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790722213368738-14927-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790722213368738-14927-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790722213368738-14927-0/krb5kdc/kudu_127.14.147.193.keytab
--principal=kudu/127.14.147.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.147.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790722213368738-14927-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.14.147.193
--webserver_port=0
--tserver_master_addrs=127.14.147.254:39875
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25234
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.147.193
--log_dir=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790722213368738-14927-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 21:58:54 UTC on ccycloud.quasar-xguoti.root.comops.site
build id 4681390
I20260929 22:51:07.101327 25234 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 22:51:07.101555 25234 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 22:51:07.106577 25240 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 22:51:07.106586 25239 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 22:51:07.106587 25242 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 22:51:07.107695 25234 server_base.cc:1034] running on GCE node
I20260929 22:51:07.108035 25234 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 22:51:07.108368 25234 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 22:51:07.109560 25234 hybrid_clock.cc:648] HybridClock initialized: now 1790722267109537 us; error 35 us; skew 500 ppm
Sep 29 22:51:07 dist-test-slave-98mk krb5kdc[25126](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790722267, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.147.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 22:51:07.112169 25234 init.cc:377] Logged in from keytab as kudu/127.14.147.193@KRBTEST.COM (short username kudu)
I20260929 22:51:07.112625 25234 webserver.cc:460] Webserver started at http://127.14.147.193:40103/ using document root <none> and password file <none>
I20260929 22:51:07.112854 25234 fs_manager.cc:359] Metadata directory not provided
I20260929 22:51:07.112903 25234 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 22:51:07.113034 25234 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 22:51:07.114085 25234 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790722213368738-14927-0/minicluster-data/ts-0/data/instance:
uuid: "f84883c2c2c4436f9000c85531038069"
format_stamp: "Formatted at 2026-09-29 22:51:07 on dist-test-slave-98mk"
server_key: "f408a63f2dcf9117566d253bc12b14f2"
server_key_iv: "53bbd646dc0593705b67f381caafe346"
server_key_version: "encryptionkey@0"
I20260929 22:51:07.114425 25234 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790722213368738-14927-0/minicluster-data/ts-0/wal/instance:
uuid: "f84883c2c2c4436f9000c85531038069"
format_stamp: "Formatted at 2026-09-29 22:51:07 on dist-test-slave-98mk"
server_key: "f408a63f2dcf9117566d253bc12b14f2"
server_key_iv: "53bbd646dc0593705b67f381caafe346"
server_key_version: "encryptionkey@0"
I20260929 22:51:07.115767 25234 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260929 22:51:07.116627 25249 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 22:51:07.116854 25234 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260929 22:51:07.116947 25234 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790722213368738-14927-0/minicluster-data/ts-0/data,/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790722213368738-14927-0/minicluster-data/ts-0/wal
uuid: "f84883c2c2c4436f9000c85531038069"
format_stamp: "Formatted at 2026-09-29 22:51:07 on dist-test-slave-98mk"
server_key: "f408a63f2dcf9117566d253bc12b14f2"
server_key_iv: "53bbd646dc0593705b67f381caafe346"
server_key_version: "encryptionkey@0"
I20260929 22:51:07.117014 25234 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790722213368738-14927-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790722213368738-14927-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790722213368738-14927-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 22:51:07.131294 25234 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 22:51:07.133502 25234 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 22:51:07.133652 25234 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 22:51:07.133883 25234 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 22:51:07.134194 25234 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 22:51:07.134225 25234 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 22:51:07.134256 25234 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 22:51:07.134289 25234 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 22:51:07.140942 25234 rpc_server.cc:307] RPC server started. Bound to: 127.14.147.193:36153
I20260929 22:51:07.140996 25363 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.147.193:36153 every 8 connection(s)
I20260929 22:51:07.141312 25234 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790722213368738-14927-0/minicluster-data/ts-0/data/info.pb
Sep 29 22:51:07 dist-test-slave-98mk krb5kdc[25126](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790722267, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.147.193@KRBTEST.COM for kudu/127.14.147.254@KRBTEST.COM
I20260929 22:51:07.149672 14927 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhWOCLk/build/release/bin/kudu as pid 25234
I20260929 22:51:07.149781 14927 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790722213368738-14927-0/minicluster-data/ts-0/wal/instance
I20260929 22:51:07.150007 14927 external_mini_cluster.cc:1428] Setting key de228c1507e5bb3d7c470f11eb013ed8
I20260929 22:51:07.150969 25212 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:51:07.142024 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.147.193:52887 (local address 127.14.147.254:39875)
0929 22:51:07.142084 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0929 22:51:07.142087 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 22:51:07.142295 (+ 208us) server_negotiation.cc:407] Connection header received
0929 22:51:07.142802 (+ 507us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 22:51:07.142803 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 22:51:07.142816 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 22:51:07.142858 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0929 22:51:07.143847 (+ 989us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:51:07.144555 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:51:07.145706 (+ 1151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:51:07.145943 (+ 237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:51:07.146013 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 22:51:07.148160 (+ 2147us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 22:51:07.148162 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 22:51:07.148163 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 22:51:07.148170 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0929 22:51:07.149497 (+ 1327us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 22:51:07.150007 (+ 510us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 22:51:07.150008 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 22:51:07.150008 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 22:51:07.150068 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 22:51:07.150337 (+ 269us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 22:51:07.150338 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 22:51:07.150338 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 22:51:07.150454 (+ 116us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 22:51:07.150510 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0929 22:51:07.150774 (+ 264us) server_negotiation.cc:299] Negotiation successful
0929 22:51:07.150878 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260929 22:51:07.151149 25366 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:51:07.141999 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.147.254:39875 (local address 127.14.147.193:52887)
0929 22:51:07.142167 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0929 22:51:07.142210 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0929 22:51:07.142698 (+ 488us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 22:51:07.142914 (+ 216us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 22:51:07.142918 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 22:51:07.143539 (+ 621us) client_negotiation.cc:189] Negotiated authn=SASL
0929 22:51:07.143746 (+ 207us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 22:51:07.143751 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 22:51:07.144713 (+ 962us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 22:51:07.144714 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 22:51:07.145570 (+ 856us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 22:51:07.145573 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 22:51:07.146016 (+ 443us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 22:51:07.146017 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 22:51:07.146124 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 22:51:07.146582 (+ 458us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 22:51:07.146590 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0929 22:51:07.148062 (+ 1472us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 22:51:07.149598 (+ 1536us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 22:51:07.149608 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 22:51:07.149610 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 22:51:07.149932 (+ 322us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 22:51:07.150124 (+ 192us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 22:51:07.150125 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 22:51:07.150125 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 22:51:07.150266 (+ 141us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 22:51:07.150510 (+ 244us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 22:51:07.150517 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 22:51:07.150657 (+ 140us) client_negotiation.cc:769] Sending connection context
0929 22:51:07.150746 (+ 89us) client_negotiation.cc:240] Negotiation successful
0929 22:51:07.150921 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":80,"threads_started":1}
I20260929 22:51:07.151578 25364 heartbeater.cc:347] Connected to a master server at 127.14.147.254:39875
I20260929 22:51:07.151661 25364 heartbeater.cc:464] Registering TS with master...
I20260929 22:51:07.151809 25364 heartbeater.cc:511] Master 127.14.147.254:39875 requested a full tablet report, sending...
I20260929 22:51:07.152362 25172 ts_manager.cc:194] Registered new tserver with Master: f84883c2c2c4436f9000c85531038069 (127.14.147.193:36153)
I20260929 22:51:07.153128 25172 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.147.193@KRBTEST.COM'} at 127.14.147.193:52887
WARNING: no policy specified for kudu/127.14.147.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.147.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.147.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790722213368738-14927-0/krb5kdc/kudu_127.14.147.194.keytab.
Entry for principal kudu/127.14.147.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790722213368738-14927-0/krb5kdc/kudu_127.14.147.194.keytab.
I20260929 22:51:07.174194 14927 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhWOCLk/build/release/bin/kudu
/tmp/dist-test-taskhWOCLk/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790722213368738-14927-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790722213368738-14927-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790722213368738-14927-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790722213368738-14927-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.147.194:0
--local_ip_for_outbound_sockets=127.14.147.194
--webserver_interface=127.14.147.194
--webserver_port=0
--tserver_master_addrs=127.14.147.254:39875
--keytab_file=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790722213368738-14927-0/krb5kdc/kudu_127.14.147.194.keytab
--principal=kudu/127.14.147.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.147.212:37831
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790722213368738-14927-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790722213368738-14927-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790722213368738-14927-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 22:51:07.254791 25369 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 22:51:07.255010 25369 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 22:51:07.255034 25369 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 22:51:07.256364 25369 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 22:51:07.256417 25369 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 22:51:07.256444 25369 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.147.194
I20260929 22:51:07.257933 25369 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.147.212:37831
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790722213368738-14927-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790722213368738-14927-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790722213368738-14927-0/krb5kdc/kudu_127.14.147.194.keytab
--principal=kudu/127.14.147.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.147.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790722213368738-14927-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.14.147.194
--webserver_port=0
--tserver_master_addrs=127.14.147.254:39875
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25369
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.147.194
--log_dir=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790722213368738-14927-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 21:58:54 UTC on ccycloud.quasar-xguoti.root.comops.site
build id 4681390
I20260929 22:51:07.258170 25369 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 22:51:07.258394 25369 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 22:51:07.263370 25374 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 22:51:07.263502 25369 server_base.cc:1034] running on GCE node
W20260929 22:51:07.263370 25375 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 22:51:07.263365 25377 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 22:51:07.263818 25369 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 22:51:07.264066 25369 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 22:51:07.265201 25369 hybrid_clock.cc:648] HybridClock initialized: now 1790722267265181 us; error 36 us; skew 500 ppm
Sep 29 22:51:07 dist-test-slave-98mk krb5kdc[25126](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790722267, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.147.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 22:51:07.267880 25369 init.cc:377] Logged in from keytab as kudu/127.14.147.194@KRBTEST.COM (short username kudu)
I20260929 22:51:07.268347 25369 webserver.cc:460] Webserver started at http://127.14.147.194:44971/ using document root <none> and password file <none>
I20260929 22:51:07.268579 25369 fs_manager.cc:359] Metadata directory not provided
I20260929 22:51:07.268625 25369 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 22:51:07.268748 25369 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 22:51:07.269806 25369 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790722213368738-14927-0/minicluster-data/ts-1/data/instance:
uuid: "b0ba1d9df1f34571b3a9a52ff1ab6af7"
format_stamp: "Formatted at 2026-09-29 22:51:07 on dist-test-slave-98mk"
server_key: "91afe827a17431e4b099f0fa9fee4197"
server_key_iv: "d72e486d4fd9442ec88291dedf94d72a"
server_key_version: "encryptionkey@0"
I20260929 22:51:07.270146 25369 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790722213368738-14927-0/minicluster-data/ts-1/wal/instance:
uuid: "b0ba1d9df1f34571b3a9a52ff1ab6af7"
format_stamp: "Formatted at 2026-09-29 22:51:07 on dist-test-slave-98mk"
server_key: "91afe827a17431e4b099f0fa9fee4197"
server_key_iv: "d72e486d4fd9442ec88291dedf94d72a"
server_key_version: "encryptionkey@0"
I20260929 22:51:07.271502 25369 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260929 22:51:07.272202 25384 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 22:51:07.272332 25369 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260929 22:51:07.272419 25369 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790722213368738-14927-0/minicluster-data/ts-1/data,/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790722213368738-14927-0/minicluster-data/ts-1/wal
uuid: "b0ba1d9df1f34571b3a9a52ff1ab6af7"
format_stamp: "Formatted at 2026-09-29 22:51:07 on dist-test-slave-98mk"
server_key: "91afe827a17431e4b099f0fa9fee4197"
server_key_iv: "d72e486d4fd9442ec88291dedf94d72a"
server_key_version: "encryptionkey@0"
I20260929 22:51:07.272491 25369 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790722213368738-14927-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790722213368738-14927-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790722213368738-14927-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 22:51:07.284708 25369 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 22:51:07.286958 25369 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 22:51:07.287093 25369 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 22:51:07.287333 25369 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 22:51:07.287696 25369 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 22:51:07.287730 25369 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 22:51:07.287755 25369 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 22:51:07.287776 25369 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 22:51:07.293499 25369 rpc_server.cc:307] RPC server started. Bound to: 127.14.147.194:43125
I20260929 22:51:07.293553 25498 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.147.194:43125 every 8 connection(s)
I20260929 22:51:07.293876 25369 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790722213368738-14927-0/minicluster-data/ts-1/data/info.pb
I20260929 22:51:07.298996 14927 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhWOCLk/build/release/bin/kudu as pid 25369
I20260929 22:51:07.299085 14927 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790722213368738-14927-0/minicluster-data/ts-1/wal/instance
I20260929 22:51:07.299274 14927 external_mini_cluster.cc:1428] Setting key bb85c20d8b5e1bce9ab3dad0b5c46bbd
Sep 29 22:51:07 dist-test-slave-98mk krb5kdc[25126](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790722267, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.147.194@KRBTEST.COM for kudu/127.14.147.254@KRBTEST.COM
I20260929 22:51:07.303391 25212 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:51:07.294613 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.147.194:38233 (local address 127.14.147.254:39875)
0929 22:51:07.294668 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0929 22:51:07.294670 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 22:51:07.294872 (+ 202us) server_negotiation.cc:407] Connection header received
0929 22:51:07.295386 (+ 514us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 22:51:07.295387 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 22:51:07.295399 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 22:51:07.295501 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0929 22:51:07.296452 (+ 951us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:51:07.297007 (+ 555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:51:07.297996 (+ 989us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:51:07.298191 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:51:07.298242 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 22:51:07.300605 (+ 2363us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 22:51:07.300606 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 22:51:07.300607 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 22:51:07.300613 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0929 22:51:07.302069 (+ 1456us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 22:51:07.302448 (+ 379us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 22:51:07.302448 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 22:51:07.302449 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 22:51:07.302498 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 22:51:07.302765 (+ 267us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 22:51:07.302766 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 22:51:07.302766 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 22:51:07.302865 (+ 99us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 22:51:07.302923 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0929 22:51:07.303203 (+ 280us) server_negotiation.cc:299] Negotiation successful
0929 22:51:07.303299 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260929 22:51:07.303601 25501 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:51:07.294578 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.147.254:39875 (local address 127.14.147.194:38233)
0929 22:51:07.294745 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0929 22:51:07.294786 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0929 22:51:07.295274 (+ 488us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 22:51:07.295571 (+ 297us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 22:51:07.295574 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 22:51:07.296145 (+ 571us) client_negotiation.cc:189] Negotiated authn=SASL
0929 22:51:07.296342 (+ 197us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 22:51:07.296346 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 22:51:07.297114 (+ 768us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 22:51:07.297115 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 22:51:07.297923 (+ 808us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 22:51:07.297926 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 22:51:07.298252 (+ 326us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 22:51:07.298253 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 22:51:07.298360 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 22:51:07.298890 (+ 530us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 22:51:07.298900 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0929 22:51:07.300504 (+ 1604us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 22:51:07.302147 (+ 1643us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 22:51:07.302154 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 22:51:07.302156 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 22:51:07.302385 (+ 229us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 22:51:07.302570 (+ 185us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 22:51:07.302570 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 22:51:07.302571 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 22:51:07.302706 (+ 135us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 22:51:07.302938 (+ 232us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 22:51:07.302944 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 22:51:07.303083 (+ 139us) client_negotiation.cc:769] Sending connection context
0929 22:51:07.303173 (+ 90us) client_negotiation.cc:240] Negotiation successful
0929 22:51:07.303336 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":80,"threads_started":1}
I20260929 22:51:07.304047 25499 heartbeater.cc:347] Connected to a master server at 127.14.147.254:39875
I20260929 22:51:07.304141 25499 heartbeater.cc:464] Registering TS with master...
I20260929 22:51:07.304318 25499 heartbeater.cc:511] Master 127.14.147.254:39875 requested a full tablet report, sending...
I20260929 22:51:07.304698 25172 ts_manager.cc:194] Registered new tserver with Master: b0ba1d9df1f34571b3a9a52ff1ab6af7 (127.14.147.194:43125)
I20260929 22:51:07.305222 25172 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.147.194@KRBTEST.COM'} at 127.14.147.194:38233
WARNING: no policy specified for kudu/127.14.147.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.147.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.147.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790722213368738-14927-0/krb5kdc/kudu_127.14.147.195.keytab.
Entry for principal kudu/127.14.147.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790722213368738-14927-0/krb5kdc/kudu_127.14.147.195.keytab.
I20260929 22:51:07.324131 14927 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhWOCLk/build/release/bin/kudu
/tmp/dist-test-taskhWOCLk/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790722213368738-14927-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790722213368738-14927-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790722213368738-14927-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790722213368738-14927-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.147.195:0
--local_ip_for_outbound_sockets=127.14.147.195
--webserver_interface=127.14.147.195
--webserver_port=0
--tserver_master_addrs=127.14.147.254:39875
--keytab_file=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790722213368738-14927-0/krb5kdc/kudu_127.14.147.195.keytab
--principal=kudu/127.14.147.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.147.212:37831
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790722213368738-14927-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790722213368738-14927-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790722213368738-14927-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 22:51:07.403834 25504 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 22:51:07.404049 25504 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 22:51:07.404073 25504 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 22:51:07.405249 25504 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 22:51:07.405290 25504 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 22:51:07.405316 25504 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.147.195
I20260929 22:51:07.406780 25504 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.147.212:37831
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790722213368738-14927-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790722213368738-14927-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790722213368738-14927-0/krb5kdc/kudu_127.14.147.195.keytab
--principal=kudu/127.14.147.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.147.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790722213368738-14927-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.14.147.195
--webserver_port=0
--tserver_master_addrs=127.14.147.254:39875
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25504
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.147.195
--log_dir=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790722213368738-14927-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 21:58:54 UTC on ccycloud.quasar-xguoti.root.comops.site
build id 4681390
I20260929 22:51:07.406991 25504 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 22:51:07.407186 25504 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 22:51:07.412128 25509 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 22:51:07.412120 25510 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 22:51:07.412115 25512 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 22:51:07.412482 25504 server_base.cc:1034] running on GCE node
I20260929 22:51:07.412672 25504 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 22:51:07.412886 25504 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 22:51:07.414047 25504 hybrid_clock.cc:648] HybridClock initialized: now 1790722267414027 us; error 31 us; skew 500 ppm
Sep 29 22:51:07 dist-test-slave-98mk krb5kdc[25126](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790722267, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.147.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 22:51:07.416771 25504 init.cc:377] Logged in from keytab as kudu/127.14.147.195@KRBTEST.COM (short username kudu)
I20260929 22:51:07.417244 25504 webserver.cc:460] Webserver started at http://127.14.147.195:38407/ using document root <none> and password file <none>
I20260929 22:51:07.417452 25504 fs_manager.cc:359] Metadata directory not provided
I20260929 22:51:07.417500 25504 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 22:51:07.417626 25504 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 22:51:07.418674 25504 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790722213368738-14927-0/minicluster-data/ts-2/data/instance:
uuid: "b11997a2717749a0a4dc8d934e5b62d4"
format_stamp: "Formatted at 2026-09-29 22:51:07 on dist-test-slave-98mk"
server_key: "ae593c0956910cb6d9424e97f6d322f7"
server_key_iv: "90c514f17b8053e92ee521aa08202ea2"
server_key_version: "encryptionkey@0"
I20260929 22:51:07.419029 25504 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790722213368738-14927-0/minicluster-data/ts-2/wal/instance:
uuid: "b11997a2717749a0a4dc8d934e5b62d4"
format_stamp: "Formatted at 2026-09-29 22:51:07 on dist-test-slave-98mk"
server_key: "ae593c0956910cb6d9424e97f6d322f7"
server_key_iv: "90c514f17b8053e92ee521aa08202ea2"
server_key_version: "encryptionkey@0"
I20260929 22:51:07.420521 25504 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260929 22:51:07.421224 25519 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 22:51:07.421378 25504 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260929 22:51:07.421459 25504 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790722213368738-14927-0/minicluster-data/ts-2/data,/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790722213368738-14927-0/minicluster-data/ts-2/wal
uuid: "b11997a2717749a0a4dc8d934e5b62d4"
format_stamp: "Formatted at 2026-09-29 22:51:07 on dist-test-slave-98mk"
server_key: "ae593c0956910cb6d9424e97f6d322f7"
server_key_iv: "90c514f17b8053e92ee521aa08202ea2"
server_key_version: "encryptionkey@0"
I20260929 22:51:07.421540 25504 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790722213368738-14927-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790722213368738-14927-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790722213368738-14927-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 22:51:07.434795 25504 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 22:51:07.437098 25504 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 22:51:07.437269 25504 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 22:51:07.437512 25504 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 22:51:07.437834 25504 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 22:51:07.437868 25504 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 22:51:07.437899 25504 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 22:51:07.437923 25504 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 22:51:07.443559 25504 rpc_server.cc:307] RPC server started. Bound to: 127.14.147.195:41617
I20260929 22:51:07.443605 25633 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.147.195:41617 every 8 connection(s)
I20260929 22:51:07.443919 25504 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790722213368738-14927-0/minicluster-data/ts-2/data/info.pb
I20260929 22:51:07.449301 14927 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhWOCLk/build/release/bin/kudu as pid 25504
I20260929 22:51:07.449391 14927 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790722213368738-14927-0/minicluster-data/ts-2/wal/instance
I20260929 22:51:07.449575 14927 external_mini_cluster.cc:1428] Setting key 847316237cbb269cf36864bddcf908dd
Sep 29 22:51:07 dist-test-slave-98mk krb5kdc[25126](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790722267, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.147.195@KRBTEST.COM for kudu/127.14.147.254@KRBTEST.COM
I20260929 22:51:07.453109 25212 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:51:07.444587 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.147.195:59843 (local address 127.14.147.254:39875)
0929 22:51:07.444644 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0929 22:51:07.444646 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 22:51:07.444843 (+ 197us) server_negotiation.cc:407] Connection header received
0929 22:51:07.445337 (+ 494us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 22:51:07.445338 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 22:51:07.445348 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 22:51:07.445416 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0929 22:51:07.446332 (+ 916us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:51:07.446835 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:51:07.447819 (+ 984us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:51:07.448004 (+ 185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:51:07.448057 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 22:51:07.450457 (+ 2400us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 22:51:07.450459 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 22:51:07.450459 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 22:51:07.450468 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0929 22:51:07.451602 (+ 1134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 22:51:07.452322 (+ 720us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 22:51:07.452323 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 22:51:07.452324 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 22:51:07.452374 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 22:51:07.452601 (+ 227us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 22:51:07.452602 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 22:51:07.452602 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 22:51:07.452704 (+ 102us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 22:51:07.452748 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0929 22:51:07.452982 (+ 234us) server_negotiation.cc:299] Negotiation successful
0929 22:51:07.453043 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260929 22:51:07.453310 25636 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:51:07.444547 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.147.254:39875 (local address 127.14.147.195:59843)
0929 22:51:07.444712 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0929 22:51:07.444756 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0929 22:51:07.445258 (+ 502us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 22:51:07.445480 (+ 222us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 22:51:07.445484 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 22:51:07.446059 (+ 575us) client_negotiation.cc:189] Negotiated authn=SASL
0929 22:51:07.446258 (+ 199us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 22:51:07.446262 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 22:51:07.446903 (+ 641us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 22:51:07.446903 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 22:51:07.447747 (+ 844us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 22:51:07.447750 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 22:51:07.448066 (+ 316us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 22:51:07.448067 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 22:51:07.448165 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 22:51:07.448661 (+ 496us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 22:51:07.448670 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0929 22:51:07.450355 (+ 1685us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 22:51:07.451714 (+ 1359us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 22:51:07.451732 (+ 18us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 22:51:07.451735 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 22:51:07.452249 (+ 514us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 22:51:07.452425 (+ 176us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 22:51:07.452425 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 22:51:07.452426 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 22:51:07.452557 (+ 131us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 22:51:07.452748 (+ 191us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 22:51:07.452751 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 22:51:07.452877 (+ 126us) client_negotiation.cc:769] Sending connection context
0929 22:51:07.452962 (+ 85us) client_negotiation.cc:240] Negotiation successful
0929 22:51:07.453091 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":84,"threads_started":1}
I20260929 22:51:07.453755 25634 heartbeater.cc:347] Connected to a master server at 127.14.147.254:39875
I20260929 22:51:07.453852 25634 heartbeater.cc:464] Registering TS with master...
I20260929 22:51:07.454064 25634 heartbeater.cc:511] Master 127.14.147.254:39875 requested a full tablet report, sending...
I20260929 22:51:07.454440 25172 ts_manager.cc:194] Registered new tserver with Master: b11997a2717749a0a4dc8d934e5b62d4 (127.14.147.195:41617)
I20260929 22:51:07.454794 25172 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.147.195@KRBTEST.COM'} at 127.14.147.195:59843
I20260929 22:51:07.461364 14927 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 22:51:07.468772 25212 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:51:07.462684 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49488 (local address 127.14.147.254:39875)
0929 22:51:07.462750 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0929 22:51:07.462753 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 22:51:07.462832 (+ 79us) server_negotiation.cc:407] Connection header received
0929 22:51:07.462884 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 22:51:07.462885 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 22:51:07.462894 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 22:51:07.462942 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0929 22:51:07.463509 (+ 567us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:51:07.463998 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:51:07.464766 (+ 768us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:51:07.464944 (+ 178us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:51:07.464989 (+ 45us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 22:51:07.466272 (+ 1283us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 22:51:07.466274 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 22:51:07.466274 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 22:51:07.466282 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0929 22:51:07.467584 (+ 1302us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 22:51:07.467920 (+ 336us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 22:51:07.467921 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 22:51:07.467921 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 22:51:07.467980 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 22:51:07.468239 (+ 259us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 22:51:07.468240 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 22:51:07.468240 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 22:51:07.468383 (+ 143us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 22:51:07.468461 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0929 22:51:07.468642 (+ 181us) server_negotiation.cc:299] Negotiation successful
0929 22:51:07.468705 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260929 22:51:07.470142 25172 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49488:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 22:51:07.471836 25172 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 22:51:07.478003 25644 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:51:07.473771 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.147.195:41617 (local address 127.0.0.1:43434)
0929 22:51:07.474139 (+ 368us) negotiation.cc:107] Waiting for socket to connect
0929 22:51:07.474180 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0929 22:51:07.474249 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 22:51:07.474460 (+ 211us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 22:51:07.474463 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 22:51:07.474466 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 22:51:07.474541 (+ 75us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 22:51:07.474543 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 22:51:07.475739 (+ 1196us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 22:51:07.475740 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 22:51:07.476918 (+ 1178us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 22:51:07.476920 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 22:51:07.477655 (+ 735us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 22:51:07.477656 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 22:51:07.477764 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 22:51:07.477779 (+ 15us) client_negotiation.cc:769] Sending connection context
0929 22:51:07.477854 (+ 75us) client_negotiation.cc:240] Negotiation successful
0929 22:51:07.477912 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":335,"mutex_wait_us":70,"thread_start_us":71,"threads_started":1}
I20260929 22:51:07.478011 25645 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:51:07.473950 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43434 (local address 127.14.147.195:41617)
0929 22:51:07.474108 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0929 22:51:07.474113 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 22:51:07.474244 (+ 131us) server_negotiation.cc:407] Connection header received
0929 22:51:07.474329 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 22:51:07.474331 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 22:51:07.474363 (+ 32us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 22:51:07.474442 (+ 79us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 22:51:07.474627 (+ 185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:51:07.475640 (+ 1013us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:51:07.476993 (+ 1353us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:51:07.477572 (+ 579us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:51:07.477647 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 22:51:07.477700 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0929 22:51:07.477873 (+ 173us) server_negotiation.cc:299] Negotiation successful
0929 22:51:07.477917 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":124,"thread_start_us":89,"threads_started":1}
I20260929 22:51:07.478370 25642 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:51:07.473776 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.147.194:43125 (local address 127.0.0.1:48024)
0929 22:51:07.474473 (+ 697us) negotiation.cc:107] Waiting for socket to connect
0929 22:51:07.474480 (+ 7us) client_negotiation.cc:174] Beginning negotiation
0929 22:51:07.474519 (+ 39us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 22:51:07.474766 (+ 247us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 22:51:07.474767 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 22:51:07.474769 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 22:51:07.474802 (+ 33us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 22:51:07.474803 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 22:51:07.476070 (+ 1267us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 22:51:07.476070 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 22:51:07.477260 (+ 1190us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 22:51:07.477262 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 22:51:07.478141 (+ 879us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 22:51:07.478142 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 22:51:07.478206 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 22:51:07.478210 (+ 4us) client_negotiation.cc:769] Sending connection context
0929 22:51:07.478252 (+ 42us) client_negotiation.cc:240] Negotiation successful
0929 22:51:07.478298 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":688,"mutex_wait_us":70,"thread_start_us":63,"threads_started":1}
I20260929 22:51:07.478384 25643 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:51:07.473771 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.147.193:36153 (local address 127.0.0.1:56820)
0929 22:51:07.474345 (+ 574us) negotiation.cc:107] Waiting for socket to connect
0929 22:51:07.474354 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0929 22:51:07.474401 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 22:51:07.474590 (+ 189us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 22:51:07.474591 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 22:51:07.474592 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 22:51:07.474649 (+ 57us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 22:51:07.474650 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 22:51:07.476062 (+ 1412us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 22:51:07.476063 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 22:51:07.477259 (+ 1196us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 22:51:07.477262 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 22:51:07.478148 (+ 886us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 22:51:07.478149 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 22:51:07.478204 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 22:51:07.478206 (+ 2us) client_negotiation.cc:769] Sending connection context
0929 22:51:07.478252 (+ 46us) client_negotiation.cc:240] Negotiation successful
0929 22:51:07.478303 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":563,"mutex_wait_us":4,"thread_start_us":71,"threads_started":1}
I20260929 22:51:07.478399 25646 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:51:07.473970 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56820 (local address 127.14.147.193:36153)
0929 22:51:07.474172 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0929 22:51:07.474177 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 22:51:07.474405 (+ 228us) server_negotiation.cc:407] Connection header received
0929 22:51:07.474491 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 22:51:07.474493 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 22:51:07.474509 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 22:51:07.474595 (+ 86us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 22:51:07.474735 (+ 140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:51:07.475895 (+ 1160us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:51:07.477341 (+ 1446us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:51:07.478000 (+ 659us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:51:07.478066 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 22:51:07.478115 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0929 22:51:07.478280 (+ 165us) server_negotiation.cc:299] Negotiation successful
0929 22:51:07.478323 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":167,"thread_start_us":91,"threads_started":1}
I20260929 22:51:07.478698 25647 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:51:07.474054 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48024 (local address 127.14.147.194:43125)
0929 22:51:07.474239 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0929 22:51:07.474243 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 22:51:07.474568 (+ 325us) server_negotiation.cc:407] Connection header received
0929 22:51:07.474609 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 22:51:07.474611 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 22:51:07.474626 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 22:51:07.474692 (+ 66us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 22:51:07.474884 (+ 192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:51:07.475977 (+ 1093us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:51:07.477353 (+ 1376us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:51:07.477948 (+ 595us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:51:07.478006 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 22:51:07.478054 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0929 22:51:07.478561 (+ 507us) server_negotiation.cc:299] Negotiation successful
0929 22:51:07.478610 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":157,"thread_start_us":73,"threads_started":1}
I20260929 22:51:07.479135 25298 tablet_service.cc:1467] Processing CreateTablet for tablet 64dc36f456474549821c1886b9d96980 (DEFAULT_TABLE table=test-table [id=633c6f6e9839408796f1526a62e4dd3f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 22:51:07.479266 25433 tablet_service.cc:1467] Processing CreateTablet for tablet 64dc36f456474549821c1886b9d96980 (DEFAULT_TABLE table=test-table [id=633c6f6e9839408796f1526a62e4dd3f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 22:51:07.479218 25568 tablet_service.cc:1467] Processing CreateTablet for tablet 64dc36f456474549821c1886b9d96980 (DEFAULT_TABLE table=test-table [id=633c6f6e9839408796f1526a62e4dd3f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 22:51:07.479409 25298 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 64dc36f456474549821c1886b9d96980. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 22:51:07.479586 25568 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 64dc36f456474549821c1886b9d96980. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 22:51:07.479586 25433 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 64dc36f456474549821c1886b9d96980. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 22:51:07.482733 25649 tablet_bootstrap.cc:492] T 64dc36f456474549821c1886b9d96980 P b11997a2717749a0a4dc8d934e5b62d4: Bootstrap starting.
I20260929 22:51:07.482924 25648 tablet_bootstrap.cc:492] T 64dc36f456474549821c1886b9d96980 P f84883c2c2c4436f9000c85531038069: Bootstrap starting.
I20260929 22:51:07.483371 25649 tablet_bootstrap.cc:654] T 64dc36f456474549821c1886b9d96980 P b11997a2717749a0a4dc8d934e5b62d4: Neither blocks nor log segments found. Creating new log.
I20260929 22:51:07.483532 25648 tablet_bootstrap.cc:654] T 64dc36f456474549821c1886b9d96980 P f84883c2c2c4436f9000c85531038069: Neither blocks nor log segments found. Creating new log.
I20260929 22:51:07.483587 25650 tablet_bootstrap.cc:492] T 64dc36f456474549821c1886b9d96980 P b0ba1d9df1f34571b3a9a52ff1ab6af7: Bootstrap starting.
I20260929 22:51:07.483969 25649 log.cc:824] T 64dc36f456474549821c1886b9d96980 P b11997a2717749a0a4dc8d934e5b62d4: Log is configured to *not* fsync() on all Append() calls
I20260929 22:51:07.484004 25648 log.cc:824] T 64dc36f456474549821c1886b9d96980 P f84883c2c2c4436f9000c85531038069: Log is configured to *not* fsync() on all Append() calls
I20260929 22:51:07.484243 25650 tablet_bootstrap.cc:654] T 64dc36f456474549821c1886b9d96980 P b0ba1d9df1f34571b3a9a52ff1ab6af7: Neither blocks nor log segments found. Creating new log.
I20260929 22:51:07.484746 25650 log.cc:824] T 64dc36f456474549821c1886b9d96980 P b0ba1d9df1f34571b3a9a52ff1ab6af7: Log is configured to *not* fsync() on all Append() calls
I20260929 22:51:07.484863 25649 tablet_bootstrap.cc:492] T 64dc36f456474549821c1886b9d96980 P b11997a2717749a0a4dc8d934e5b62d4: No bootstrap required, opened a new log
I20260929 22:51:07.484865 25648 tablet_bootstrap.cc:492] T 64dc36f456474549821c1886b9d96980 P f84883c2c2c4436f9000c85531038069: No bootstrap required, opened a new log
I20260929 22:51:07.484932 25648 ts_tablet_manager.cc:1397] T 64dc36f456474549821c1886b9d96980 P f84883c2c2c4436f9000c85531038069: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260929 22:51:07.484932 25649 ts_tablet_manager.cc:1397] T 64dc36f456474549821c1886b9d96980 P b11997a2717749a0a4dc8d934e5b62d4: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260929 22:51:07.486480 25648 raft_consensus.cc:348] T 64dc36f456474549821c1886b9d96980 P f84883c2c2c4436f9000c85531038069 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f84883c2c2c4436f9000c85531038069" member_type: VOTER last_known_addr { host: "127.14.147.193" port: 36153 } } peers { permanent_uuid: "b0ba1d9df1f34571b3a9a52ff1ab6af7" member_type: VOTER last_known_addr { host: "127.14.147.194" port: 43125 } } peers { permanent_uuid: "b11997a2717749a0a4dc8d934e5b62d4" member_type: VOTER last_known_addr { host: "127.14.147.195" port: 41617 } }
I20260929 22:51:07.486480 25649 raft_consensus.cc:348] T 64dc36f456474549821c1886b9d96980 P b11997a2717749a0a4dc8d934e5b62d4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f84883c2c2c4436f9000c85531038069" member_type: VOTER last_known_addr { host: "127.14.147.193" port: 36153 } } peers { permanent_uuid: "b0ba1d9df1f34571b3a9a52ff1ab6af7" member_type: VOTER last_known_addr { host: "127.14.147.194" port: 43125 } } peers { permanent_uuid: "b11997a2717749a0a4dc8d934e5b62d4" member_type: VOTER last_known_addr { host: "127.14.147.195" port: 41617 } }
I20260929 22:51:07.486619 25649 raft_consensus.cc:374] T 64dc36f456474549821c1886b9d96980 P b11997a2717749a0a4dc8d934e5b62d4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 22:51:07.486621 25648 raft_consensus.cc:374] T 64dc36f456474549821c1886b9d96980 P f84883c2c2c4436f9000c85531038069 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 22:51:07.486645 25649 raft_consensus.cc:729] T 64dc36f456474549821c1886b9d96980 P b11997a2717749a0a4dc8d934e5b62d4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b11997a2717749a0a4dc8d934e5b62d4, State: Initialized, Role: FOLLOWER
I20260929 22:51:07.486652 25648 raft_consensus.cc:729] T 64dc36f456474549821c1886b9d96980 P f84883c2c2c4436f9000c85531038069 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f84883c2c2c4436f9000c85531038069, State: Initialized, Role: FOLLOWER
I20260929 22:51:07.486753 25649 consensus_queue.cc:260] T 64dc36f456474549821c1886b9d96980 P b11997a2717749a0a4dc8d934e5b62d4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f84883c2c2c4436f9000c85531038069" member_type: VOTER last_known_addr { host: "127.14.147.193" port: 36153 } } peers { permanent_uuid: "b0ba1d9df1f34571b3a9a52ff1ab6af7" member_type: VOTER last_known_addr { host: "127.14.147.194" port: 43125 } } peers { permanent_uuid: "b11997a2717749a0a4dc8d934e5b62d4" member_type: VOTER last_known_addr { host: "127.14.147.195" port: 41617 } }
I20260929 22:51:07.486799 25648 consensus_queue.cc:260] T 64dc36f456474549821c1886b9d96980 P f84883c2c2c4436f9000c85531038069 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f84883c2c2c4436f9000c85531038069" member_type: VOTER last_known_addr { host: "127.14.147.193" port: 36153 } } peers { permanent_uuid: "b0ba1d9df1f34571b3a9a52ff1ab6af7" member_type: VOTER last_known_addr { host: "127.14.147.194" port: 43125 } } peers { permanent_uuid: "b11997a2717749a0a4dc8d934e5b62d4" member_type: VOTER last_known_addr { host: "127.14.147.195" port: 41617 } }
I20260929 22:51:07.487028 25649 ts_tablet_manager.cc:1428] T 64dc36f456474549821c1886b9d96980 P b11997a2717749a0a4dc8d934e5b62d4: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260929 22:51:07.487040 25634 heartbeater.cc:503] Master 127.14.147.254:39875 was elected leader, sending a full tablet report...
I20260929 22:51:07.487112 25364 heartbeater.cc:503] Master 127.14.147.254:39875 was elected leader, sending a full tablet report...
I20260929 22:51:07.487130 25648 ts_tablet_manager.cc:1428] T 64dc36f456474549821c1886b9d96980 P f84883c2c2c4436f9000c85531038069: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260929 22:51:07.487787 25650 tablet_bootstrap.cc:492] T 64dc36f456474549821c1886b9d96980 P b0ba1d9df1f34571b3a9a52ff1ab6af7: No bootstrap required, opened a new log
I20260929 22:51:07.487874 25650 ts_tablet_manager.cc:1397] T 64dc36f456474549821c1886b9d96980 P b0ba1d9df1f34571b3a9a52ff1ab6af7: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260929 22:51:07.489391 25650 raft_consensus.cc:348] T 64dc36f456474549821c1886b9d96980 P b0ba1d9df1f34571b3a9a52ff1ab6af7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f84883c2c2c4436f9000c85531038069" member_type: VOTER last_known_addr { host: "127.14.147.193" port: 36153 } } peers { permanent_uuid: "b0ba1d9df1f34571b3a9a52ff1ab6af7" member_type: VOTER last_known_addr { host: "127.14.147.194" port: 43125 } } peers { permanent_uuid: "b11997a2717749a0a4dc8d934e5b62d4" member_type: VOTER last_known_addr { host: "127.14.147.195" port: 41617 } }
I20260929 22:51:07.489532 25650 raft_consensus.cc:374] T 64dc36f456474549821c1886b9d96980 P b0ba1d9df1f34571b3a9a52ff1ab6af7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 22:51:07.489554 25650 raft_consensus.cc:729] T 64dc36f456474549821c1886b9d96980 P b0ba1d9df1f34571b3a9a52ff1ab6af7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b0ba1d9df1f34571b3a9a52ff1ab6af7, State: Initialized, Role: FOLLOWER
I20260929 22:51:07.489645 25650 consensus_queue.cc:260] T 64dc36f456474549821c1886b9d96980 P b0ba1d9df1f34571b3a9a52ff1ab6af7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f84883c2c2c4436f9000c85531038069" member_type: VOTER last_known_addr { host: "127.14.147.193" port: 36153 } } peers { permanent_uuid: "b0ba1d9df1f34571b3a9a52ff1ab6af7" member_type: VOTER last_known_addr { host: "127.14.147.194" port: 43125 } } peers { permanent_uuid: "b11997a2717749a0a4dc8d934e5b62d4" member_type: VOTER last_known_addr { host: "127.14.147.195" port: 41617 } }
I20260929 22:51:07.489888 25650 ts_tablet_manager.cc:1428] T 64dc36f456474549821c1886b9d96980 P b0ba1d9df1f34571b3a9a52ff1ab6af7: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260929 22:51:07.489916 25499 heartbeater.cc:503] Master 127.14.147.254:39875 was elected leader, sending a full tablet report...
I20260929 22:51:07.497116 25655 raft_consensus.cc:482] T 64dc36f456474549821c1886b9d96980 P f84883c2c2c4436f9000c85531038069 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 22:51:07.497242 25655 raft_consensus.cc:504] T 64dc36f456474549821c1886b9d96980 P f84883c2c2c4436f9000c85531038069 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f84883c2c2c4436f9000c85531038069" member_type: VOTER last_known_addr { host: "127.14.147.193" port: 36153 } } peers { permanent_uuid: "b0ba1d9df1f34571b3a9a52ff1ab6af7" member_type: VOTER last_known_addr { host: "127.14.147.194" port: 43125 } } peers { permanent_uuid: "b11997a2717749a0a4dc8d934e5b62d4" member_type: VOTER last_known_addr { host: "127.14.147.195" port: 41617 } }
I20260929 22:51:07.497581 25655 leader_election.cc:288] T 64dc36f456474549821c1886b9d96980 P f84883c2c2c4436f9000c85531038069 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b0ba1d9df1f34571b3a9a52ff1ab6af7 (127.14.147.194:43125), b11997a2717749a0a4dc8d934e5b62d4 (127.14.147.195:41617)
I20260929 22:51:07.500281 25366 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:51:07.497732 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.147.195:41617 (local address 127.14.147.193:38253)
0929 22:51:07.497810 (+ 78us) negotiation.cc:107] Waiting for socket to connect
0929 22:51:07.497821 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0929 22:51:07.497857 (+ 36us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 22:51:07.497995 (+ 138us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 22:51:07.497995 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 22:51:07.497997 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 22:51:07.498061 (+ 64us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 22:51:07.498063 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 22:51:07.498654 (+ 591us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 22:51:07.498655 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 22:51:07.499506 (+ 851us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 22:51:07.499508 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 22:51:07.500027 (+ 519us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 22:51:07.500028 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 22:51:07.500114 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 22:51:07.500117 (+ 3us) client_negotiation.cc:769] Sending connection context
0929 22:51:07.500162 (+ 45us) client_negotiation.cc:240] Negotiation successful
0929 22:51:07.500204 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":59,"mutex_wait_us":40}
I20260929 22:51:07.500281 25645 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:51:07.497834 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.147.193:38253 (local address 127.14.147.195:41617)
0929 22:51:07.497889 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0929 22:51:07.497891 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 22:51:07.497899 (+ 8us) server_negotiation.cc:407] Connection header received
0929 22:51:07.497924 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 22:51:07.497925 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 22:51:07.497932 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 22:51:07.497983 (+ 51us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 22:51:07.498113 (+ 130us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:51:07.498588 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:51:07.499577 (+ 989us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:51:07.499960 (+ 383us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:51:07.500020 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 22:51:07.500029 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0929 22:51:07.500184 (+ 155us) server_negotiation.cc:299] Negotiation successful
0929 22:51:07.500219 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260929 22:51:07.500475 25647 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:51:07.497810 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.147.193:58741 (local address 127.14.147.194:43125)
0929 22:51:07.497849 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0929 22:51:07.497851 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 22:51:07.498026 (+ 175us) server_negotiation.cc:407] Connection header received
0929 22:51:07.498074 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 22:51:07.498075 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 22:51:07.498084 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 22:51:07.498136 (+ 52us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 22:51:07.498257 (+ 121us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:51:07.498722 (+ 465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:51:07.499664 (+ 942us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:51:07.500053 (+ 389us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:51:07.500113 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 22:51:07.500121 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0929 22:51:07.500347 (+ 226us) server_negotiation.cc:299] Negotiation successful
0929 22:51:07.500399 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260929 22:51:07.500624 25657 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:51:07.497739 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.147.194:43125 (local address 127.14.147.193:58741)
0929 22:51:07.497959 (+ 220us) negotiation.cc:107] Waiting for socket to connect
0929 22:51:07.497968 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0929 22:51:07.498023 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 22:51:07.498139 (+ 116us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 22:51:07.498140 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 22:51:07.498141 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 22:51:07.498207 (+ 66us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 22:51:07.498209 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 22:51:07.498788 (+ 579us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 22:51:07.498789 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 22:51:07.499591 (+ 802us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 22:51:07.499594 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 22:51:07.500121 (+ 527us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 22:51:07.500122 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 22:51:07.500249 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 22:51:07.500254 (+ 5us) client_negotiation.cc:769] Sending connection context
0929 22:51:07.500323 (+ 69us) client_negotiation.cc:240] Negotiation successful
0929 22:51:07.500497 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":201,"mutex_wait_us":46,"thread_start_us":108,"threads_started":1}
I20260929 22:51:07.500607 25588 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "64dc36f456474549821c1886b9d96980" candidate_uuid: "f84883c2c2c4436f9000c85531038069" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b11997a2717749a0a4dc8d934e5b62d4" is_pre_election: true
I20260929 22:51:07.500775 25588 raft_consensus.cc:2454] T 64dc36f456474549821c1886b9d96980 P b11997a2717749a0a4dc8d934e5b62d4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f84883c2c2c4436f9000c85531038069 in term 0.
I20260929 22:51:07.500931 25453 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "64dc36f456474549821c1886b9d96980" candidate_uuid: "f84883c2c2c4436f9000c85531038069" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b0ba1d9df1f34571b3a9a52ff1ab6af7" is_pre_election: true
I20260929 22:51:07.500994 25252 leader_election.cc:302] T 64dc36f456474549821c1886b9d96980 P f84883c2c2c4436f9000c85531038069 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b11997a2717749a0a4dc8d934e5b62d4, f84883c2c2c4436f9000c85531038069; no voters:
I20260929 22:51:07.501078 25453 raft_consensus.cc:2454] T 64dc36f456474549821c1886b9d96980 P b0ba1d9df1f34571b3a9a52ff1ab6af7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f84883c2c2c4436f9000c85531038069 in term 0.
I20260929 22:51:07.501096 25655 raft_consensus.cc:2781] T 64dc36f456474549821c1886b9d96980 P f84883c2c2c4436f9000c85531038069 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 22:51:07.501147 25655 raft_consensus.cc:482] T 64dc36f456474549821c1886b9d96980 P f84883c2c2c4436f9000c85531038069 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 22:51:07.501168 25655 raft_consensus.cc:3037] T 64dc36f456474549821c1886b9d96980 P f84883c2c2c4436f9000c85531038069 [term 0 FOLLOWER]: Advancing to term 1
I20260929 22:51:07.501880 25655 raft_consensus.cc:504] T 64dc36f456474549821c1886b9d96980 P f84883c2c2c4436f9000c85531038069 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f84883c2c2c4436f9000c85531038069" member_type: VOTER last_known_addr { host: "127.14.147.193" port: 36153 } } peers { permanent_uuid: "b0ba1d9df1f34571b3a9a52ff1ab6af7" member_type: VOTER last_known_addr { host: "127.14.147.194" port: 43125 } } peers { permanent_uuid: "b11997a2717749a0a4dc8d934e5b62d4" member_type: VOTER last_known_addr { host: "127.14.147.195" port: 41617 } }
I20260929 22:51:07.502019 25655 leader_election.cc:288] T 64dc36f456474549821c1886b9d96980 P f84883c2c2c4436f9000c85531038069 [CANDIDATE]: Term 1 election: Requested vote from peers b0ba1d9df1f34571b3a9a52ff1ab6af7 (127.14.147.194:43125), b11997a2717749a0a4dc8d934e5b62d4 (127.14.147.195:41617)
I20260929 22:51:07.502146 25588 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "64dc36f456474549821c1886b9d96980" candidate_uuid: "f84883c2c2c4436f9000c85531038069" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b11997a2717749a0a4dc8d934e5b62d4"
I20260929 22:51:07.502167 25453 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "64dc36f456474549821c1886b9d96980" candidate_uuid: "f84883c2c2c4436f9000c85531038069" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b0ba1d9df1f34571b3a9a52ff1ab6af7"
I20260929 22:51:07.502225 25588 raft_consensus.cc:3037] T 64dc36f456474549821c1886b9d96980 P b11997a2717749a0a4dc8d934e5b62d4 [term 0 FOLLOWER]: Advancing to term 1
I20260929 22:51:07.502233 25453 raft_consensus.cc:3037] T 64dc36f456474549821c1886b9d96980 P b0ba1d9df1f34571b3a9a52ff1ab6af7 [term 0 FOLLOWER]: Advancing to term 1
I20260929 22:51:07.502914 25588 raft_consensus.cc:2454] T 64dc36f456474549821c1886b9d96980 P b11997a2717749a0a4dc8d934e5b62d4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f84883c2c2c4436f9000c85531038069 in term 1.
I20260929 22:51:07.502916 25453 raft_consensus.cc:2454] T 64dc36f456474549821c1886b9d96980 P b0ba1d9df1f34571b3a9a52ff1ab6af7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f84883c2c2c4436f9000c85531038069 in term 1.
I20260929 22:51:07.503084 25253 leader_election.cc:302] T 64dc36f456474549821c1886b9d96980 P f84883c2c2c4436f9000c85531038069 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b0ba1d9df1f34571b3a9a52ff1ab6af7, f84883c2c2c4436f9000c85531038069; no voters:
I20260929 22:51:07.503196 25655 raft_consensus.cc:2781] T 64dc36f456474549821c1886b9d96980 P f84883c2c2c4436f9000c85531038069 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 22:51:07.503353 25655 raft_consensus.cc:686] T 64dc36f456474549821c1886b9d96980 P f84883c2c2c4436f9000c85531038069 [term 1 LEADER]: Becoming Leader. State: Replica: f84883c2c2c4436f9000c85531038069, State: Running, Role: LEADER
I20260929 22:51:07.503455 25655 consensus_queue.cc:237] T 64dc36f456474549821c1886b9d96980 P f84883c2c2c4436f9000c85531038069 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f84883c2c2c4436f9000c85531038069" member_type: VOTER last_known_addr { host: "127.14.147.193" port: 36153 } } peers { permanent_uuid: "b0ba1d9df1f34571b3a9a52ff1ab6af7" member_type: VOTER last_known_addr { host: "127.14.147.194" port: 43125 } } peers { permanent_uuid: "b11997a2717749a0a4dc8d934e5b62d4" member_type: VOTER last_known_addr { host: "127.14.147.195" port: 41617 } }
I20260929 22:51:07.504209 25172 catalog_manager.cc:5495] T 64dc36f456474549821c1886b9d96980 P f84883c2c2c4436f9000c85531038069 reported cstate change: term changed from 0 to 1, leader changed from <none> to f84883c2c2c4436f9000c85531038069 (127.14.147.193). New cstate: current_term: 1 leader_uuid: "f84883c2c2c4436f9000c85531038069" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f84883c2c2c4436f9000c85531038069" member_type: VOTER last_known_addr { host: "127.14.147.193" port: 36153 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b0ba1d9df1f34571b3a9a52ff1ab6af7" member_type: VOTER last_known_addr { host: "127.14.147.194" port: 43125 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b11997a2717749a0a4dc8d934e5b62d4" member_type: VOTER last_known_addr { host: "127.14.147.195" port: 41617 } health_report { overall_health: UNKNOWN } } }
I20260929 22:51:07.516669 25646 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:51:07.513558 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56834 (local address 127.14.147.193:36153)
0929 22:51:07.513601 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0929 22:51:07.513618 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0929 22:51:07.513626 (+ 8us) server_negotiation.cc:407] Connection header received
0929 22:51:07.513668 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 22:51:07.513669 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 22:51:07.513691 (+ 22us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 22:51:07.513761 (+ 70us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 22:51:07.513960 (+ 199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:51:07.514762 (+ 802us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:51:07.515829 (+ 1067us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:51:07.516085 (+ 256us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:51:07.516138 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 22:51:07.516306 (+ 168us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 22:51:07.516428 (+ 122us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 22:51:07.516503 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0929 22:51:07.516540 (+ 37us) server_negotiation.cc:299] Negotiation successful
0929 22:51:07.516562 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260929 22:51:07.518431 25453 raft_consensus.cc:1261] T 64dc36f456474549821c1886b9d96980 P b0ba1d9df1f34571b3a9a52ff1ab6af7 [term 1 FOLLOWER]: Refusing update from remote peer f84883c2c2c4436f9000c85531038069: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 22:51:07.518453 25588 raft_consensus.cc:1261] T 64dc36f456474549821c1886b9d96980 P b11997a2717749a0a4dc8d934e5b62d4 [term 1 FOLLOWER]: Refusing update from remote peer f84883c2c2c4436f9000c85531038069: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 22:51:07.518838 25655 consensus_queue.cc:1035] T 64dc36f456474549821c1886b9d96980 P f84883c2c2c4436f9000c85531038069 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b0ba1d9df1f34571b3a9a52ff1ab6af7" member_type: VOTER last_known_addr { host: "127.14.147.194" port: 43125 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 22:51:07.518998 25658 consensus_queue.cc:1035] T 64dc36f456474549821c1886b9d96980 P f84883c2c2c4436f9000c85531038069 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b11997a2717749a0a4dc8d934e5b62d4" member_type: VOTER last_known_addr { host: "127.14.147.195" port: 41617 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 22:51:07.521409 25661 mvcc.cc:205] Tried to move back new op lower bound from 7334798407752826880 to 7334798407694811136. Current Snapshot: MvccSnapshot[applied={T|T < 7334798407752826880}]
I20260929 22:51:07.521752 25662 mvcc.cc:205] Tried to move back new op lower bound from 7334798407752826880 to 7334798407694811136. Current Snapshot: MvccSnapshot[applied={T|T < 7334798407752826880}]
I20260929 22:51:07.522114 25663 mvcc.cc:205] Tried to move back new op lower bound from 7334798407752826880 to 7334798407694811136. Current Snapshot: MvccSnapshot[applied={T|T < 7334798407752826880}]
I20260929 22:51:07.523849 25172 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49488:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 22:51:07.523959 25172 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49488:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 22:51:07.524753 25172 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 51708af5c8894488ad0edde82284d09c: Sending DeleteTablet for 3 replicas of tablet 64dc36f456474549821c1886b9d96980
I20260929 22:51:07.525112 25568 tablet_service.cc:1514] Processing DeleteTablet for tablet 64dc36f456474549821c1886b9d96980 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 22:51:07 UTC) from {username='kudu', principal='kudu/127.14.147.254@KRBTEST.COM'} at 127.0.0.1:43434
I20260929 22:51:07.525136 25298 tablet_service.cc:1514] Processing DeleteTablet for tablet 64dc36f456474549821c1886b9d96980 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 22:51:07 UTC) from {username='kudu', principal='kudu/127.14.147.254@KRBTEST.COM'} at 127.0.0.1:56820
I20260929 22:51:07.525152 25433 tablet_service.cc:1514] Processing DeleteTablet for tablet 64dc36f456474549821c1886b9d96980 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 22:51:07 UTC) from {username='kudu', principal='kudu/127.14.147.254@KRBTEST.COM'} at 127.0.0.1:48024
I20260929 22:51:07.525386 25672 tablet_replica.cc:330] T 64dc36f456474549821c1886b9d96980 P b11997a2717749a0a4dc8d934e5b62d4: stopping tablet replica
I20260929 22:51:07.525457 14927 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhWOCLk/build/release/bin/kudu with pid 25234
I20260929 22:51:07.525489 25672 raft_consensus.cc:2229] T 64dc36f456474549821c1886b9d96980 P b11997a2717749a0a4dc8d934e5b62d4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 22:51:07.525588 25672 raft_consensus.cc:2258] T 64dc36f456474549821c1886b9d96980 P b11997a2717749a0a4dc8d934e5b62d4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 22:51:07.526654 25672 ts_tablet_manager.cc:1905] T 64dc36f456474549821c1886b9d96980 P b11997a2717749a0a4dc8d934e5b62d4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 22:51:07.528357 25673 tablet_replica.cc:330] T 64dc36f456474549821c1886b9d96980 P b0ba1d9df1f34571b3a9a52ff1ab6af7: stopping tablet replica
I20260929 22:51:07.528496 25673 raft_consensus.cc:2229] T 64dc36f456474549821c1886b9d96980 P b0ba1d9df1f34571b3a9a52ff1ab6af7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 22:51:07.528617 25673 raft_consensus.cc:2258] T 64dc36f456474549821c1886b9d96980 P b0ba1d9df1f34571b3a9a52ff1ab6af7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 22:51:07.528635 25672 ts_tablet_manager.cc:1918] T 64dc36f456474549821c1886b9d96980 P b11997a2717749a0a4dc8d934e5b62d4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 22:51:07.528704 25672 log.cc:1196] T 64dc36f456474549821c1886b9d96980 P b11997a2717749a0a4dc8d934e5b62d4: Deleting WAL directory at /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790722213368738-14927-0/minicluster-data/ts-2/wal/wals/64dc36f456474549821c1886b9d96980
I20260929 22:51:07.528995 25672 ts_tablet_manager.cc:1939] T 64dc36f456474549821c1886b9d96980 P b11997a2717749a0a4dc8d934e5b62d4: Deleting consensus metadata
I20260929 22:51:07.529091 25673 ts_tablet_manager.cc:1905] T 64dc36f456474549821c1886b9d96980 P b0ba1d9df1f34571b3a9a52ff1ab6af7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 22:51:07.529410 25157 catalog_manager.cc:4841] TS b11997a2717749a0a4dc8d934e5b62d4 (127.14.147.195:41617): tablet 64dc36f456474549821c1886b9d96980 (table test-table [id=633c6f6e9839408796f1526a62e4dd3f]) successfully deleted
I20260929 22:51:07.530566 25673 ts_tablet_manager.cc:1918] T 64dc36f456474549821c1886b9d96980 P b0ba1d9df1f34571b3a9a52ff1ab6af7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 22:51:07.530644 25673 log.cc:1196] T 64dc36f456474549821c1886b9d96980 P b0ba1d9df1f34571b3a9a52ff1ab6af7: Deleting WAL directory at /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790722213368738-14927-0/minicluster-data/ts-1/wal/wals/64dc36f456474549821c1886b9d96980
I20260929 22:51:07.530892 25673 ts_tablet_manager.cc:1939] T 64dc36f456474549821c1886b9d96980 P b0ba1d9df1f34571b3a9a52ff1ab6af7: Deleting consensus metadata
W20260929 22:51:07.531221 25156 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.14.147.193:36153 (error 108)
I20260929 22:51:07.531518 25158 catalog_manager.cc:4841] TS b0ba1d9df1f34571b3a9a52ff1ab6af7 (127.14.147.194:43125): tablet 64dc36f456474549821c1886b9d96980 (table test-table [id=633c6f6e9839408796f1526a62e4dd3f]) successfully deleted
I20260929 22:51:07.531692 25644 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:51:07.531574 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.147.193:36153 (local address 127.0.0.1:56846)
0929 22:51:07.531627 (+ 53us) negotiation.cc:107] Waiting for socket to connect
0929 22:51:07.531658 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.147.193:36153: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
W20260929 22:51:07.531821 25156 catalog_manager.cc:4584] TS f84883c2c2c4436f9000c85531038069 (127.14.147.193:36153): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 64dc36f456474549821c1886b9d96980: Network error: Client connection negotiation failed: client connection to 127.14.147.193:36153: connect: Connection refused (error 111)
I20260929 22:51:07.531901 14927 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhWOCLk/build/release/bin/kudu with pid 25369
I20260929 22:51:07.536461 14927 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhWOCLk/build/release/bin/kudu with pid 25504
I20260929 22:51:07.540551 14927 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhWOCLk/build/release/bin/kudu with pid 25139
2026-09-29T22:51:07Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationSecureCluster (1782 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegration
Loading random data
Initializing database '/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790722213368738-14927-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 22:51:07 dist-test-slave-98mk krb5kdc[25677](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 22:51:07 dist-test-slave-98mk krb5kdc[25677](info): set up 2 sockets
Sep 29 22:51:07 dist-test-slave-98mk krb5kdc[25677](info): commencing operation
krb5kdc: starting...
W20260929 22:51:08.496706 14927 mini_kdc.cc:121] Time spent starting KDC: real 0.937s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790722213368738-14927-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790722213368738-14927-0/krb5kdc/test-user.keytab.
Sep 29 22:51:08 dist-test-slave-98mk krb5kdc[25677](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790722268, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T22:51:08Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T22:51:08Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.14.147.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.14.147.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.14.147.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790722213368738-14927-0/krb5kdc/rangeradmin_127.14.147.212@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.14.147.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790722213368738-14927-0/krb5kdc/rangeradmin_127.14.147.212@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.14.147.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.14.147.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.14.147.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790722213368738-14927-0/krb5kdc/rangerlookup_127.14.147.212@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.14.147.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790722213368738-14927-0/krb5kdc/rangerlookup_127.14.147.212@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.14.147.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.14.147.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.14.147.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790722213368738-14927-0/krb5kdc/HTTP_127.14.147.212@KRBTEST.COM.keytab.
Entry for principal HTTP/127.14.147.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790722213368738-14927-0/krb5kdc/HTTP_127.14.147.212@KRBTEST.COM.keytab.
I20260929 22:51:08.653689 14927 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790722213368738-14927-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskhWOCLk/build/release/bin/postgres/pg_ctl -D /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790722213368738-14927-0/minicluster-data/postgres -l logfile start
2026-09-29 22:51:11.077 UTC [25707] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-09-29 22:51:11.078 UTC [25707] LOG: listening on IPv4 address "127.14.147.212", port 59701
2026-09-29 22:51:11.082 UTC [25707] LOG: listening on Unix socket "/tmp/.s.PGSQL.59701"
2026-09-29 22:51:11.103 UTC [25710] LOG: database system was shut down at 2026-09-29 22:51:09 UTC
2026-09-29 22:51:11.107 UTC [25707] LOG: database system is ready to accept connections
I20260929 22:51:11.972788 14927 mini_postgres.cc:96] Postgres bound to 59701
2026-09-29 22:51:11.979 UTC [25718] FATAL: database "slave" does not exist
127.14.147.212:59701 - accepting connections
I20260929 22:51:11.980538 14927 mini_ranger.cc:161] Starting Ranger...
I20260929 22:51:11.995839 14927 mini_ranger.cc:84] Created miniranger Postgres user
I20260929 22:51:12.465416 14927 mini_ranger.cc:87] Created ranger Postgres database
I20260929 22:51:12.465538 14927 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790722213368738-14927-0/minicluster-data/ranger-admin
2026-09-29 22:51:12,865 [I] DB FLAVOR :POSTGRES
2026-09-29 22:51:12,866 [I] --------- Verifying Ranger DB connection ---------
2026-09-29 22:51:12,866 [I] Checking connection..
2026-09-29 22:51:12,866 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskhWOCLk/build/release/bin/postgresql.jar:/tmp/dist-test-taskhWOCLk/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.14.147.212:59701/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-09-29 22:51:13,198 [I] Checking connection passed.
2026-09-29 22:51:13,199 [I] --------- Verifying version history table ---------
2026-09-29 22:51:13,199 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskhWOCLk/build/release/bin/postgresql.jar:/tmp/dist-test-taskhWOCLk/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.14.147.212:59701/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-29 22:51:13,459 [I] Table x_db_version_h does not exist in database ranger
2026-09-29 22:51:13,459 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskhWOCLk/build/release/bin/postgresql.jar:/tmp/dist-test-taskhWOCLk/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.14.147.212:59701/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-29 22:51:13,718 [I] Table x_db_version_h does not exist in database ranger
2026-09-29 22:51:13,718 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-09-29 22:51:13,718 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskhWOCLk/build/release/bin/postgresql.jar:/tmp/dist-test-taskhWOCLk/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.14.147.212:59701/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskhWOCLk/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-09-29 22:51:13.957 UTC [25782] WARNING: there is no transaction in progress
2026-09-29 22:51:13,963 [I] create_dbversion_catalog.sql file imported successfully
2026-09-29 22:51:13,964 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskhWOCLk/build/release/bin/postgresql.jar:/tmp/dist-test-taskhWOCLk/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.14.147.212:59701/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-29 22:51:14,209 [I] Table x_db_version_h already exists in database 'ranger'
2026-09-29 22:51:14,210 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-29 22:51:14,210 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskhWOCLk/build/release/bin/postgresql.jar:/tmp/dist-test-taskhWOCLk/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.14.147.212:59701/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-09-29 22:51:14,449 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskhWOCLk/build/release/bin/postgresql.jar:/tmp/dist-test-taskhWOCLk/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.14.147.212:59701/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-09-29 22:51:14,695 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskhWOCLk/build/release/bin/postgresql.jar:/tmp/dist-test-taskhWOCLk/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.14.147.212:59701/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-98mk.c.gcp-upstream.internal','N') ;"
2026-09-29 22:51:14,940 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskhWOCLk/build/release/bin/postgresql.jar:/tmp/dist-test-taskhWOCLk/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.14.147.212:59701/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-29 22:51:15,211 [I] Table x_portal_user does not exist in database ranger
2026-09-29 22:51:15,212 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskhWOCLk/build/release/bin/postgresql.jar:/tmp/dist-test-taskhWOCLk/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.14.147.212:59701/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-29 22:51:15,476 [I] Table x_policy_ref_group does not exist in database ranger
2026-09-29 22:51:15,476 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskhWOCLk/build/release/bin/postgresql.jar:/tmp/dist-test-taskhWOCLk/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.14.147.212:59701/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-29 22:51:15,721 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskhWOCLk/build/release/bin/postgresql.jar:/tmp/dist-test-taskhWOCLk/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.14.147.212:59701/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-29 22:51:15,974 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-09-29 22:51:15,975 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskhWOCLk/build/release/bin/postgresql.jar:/tmp/dist-test-taskhWOCLk/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.14.147.212:59701/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskhWOCLk/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-09-29 22:51:16.957 UTC [25892] WARNING: there is no transaction in progress
2026-09-29 22:51:16.974 UTC [25892] WARNING: there is no transaction in progress
2026-09-29 22:51:16.990 UTC [25892] WARNING: there is no transaction in progress
2026-09-29 22:51:17.008 UTC [25892] WARNING: there is no transaction in progress
2026-09-29 22:51:17.027 UTC [25892] WARNING: there is no transaction in progress
2026-09-29 22:51:17.045 UTC [25892] WARNING: there is no transaction in progress
2026-09-29 22:51:17.105 UTC [25892] WARNING: there is no transaction in progress
2026-09-29 22:51:17.112 UTC [25892] WARNING: there is no transaction in progress
2026-09-29 22:51:17.126 UTC [25892] WARNING: there is no transaction in progress
2026-09-29 22:51:17.137 UTC [25892] WARNING: there is no transaction in progress
2026-09-29 22:51:17.149 UTC [25892] WARNING: there is no transaction in progress
2026-09-29 22:51:17.163 UTC [25892] WARNING: there is no transaction in progress
2026-09-29 22:51:17.173 UTC [25892] WARNING: there is no transaction in progress
2026-09-29 22:51:17.179 UTC [25892] WARNING: there is no transaction in progress
2026-09-29 22:51:17.185 UTC [25892] WARNING: there is no transaction in progress
2026-09-29 22:51:17.193 UTC [25892] WARNING: there is no transaction in progress
2026-09-29 22:51:17,840 [I] ranger_core_db_postgres.sql file imported successfully
2026-09-29 22:51:17,840 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskhWOCLk/build/release/bin/postgresql.jar:/tmp/dist-test-taskhWOCLk/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.14.147.212:59701/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-09-29 22:51:18,083 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-09-29 22:51:18,084 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskhWOCLk/build/release/bin/postgresql.jar:/tmp/dist-test-taskhWOCLk/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.14.147.212:59701/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-29 22:51:18,347 [I] Table x_portal_user already exists in database 'ranger'
2026-09-29 22:51:18,348 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskhWOCLk/build/release/bin/postgresql.jar:/tmp/dist-test-taskhWOCLk/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.14.147.212:59701/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-29 22:51:18,603 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-09-29 22:51:18,603 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskhWOCLk/build/release/bin/postgresql.jar:/tmp/dist-test-taskhWOCLk/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.14.147.212:59701/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-29 22:51:18,849 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskhWOCLk/build/release/bin/postgresql.jar:/tmp/dist-test-taskhWOCLk/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.14.147.212:59701/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-29 22:51:19,100 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskhWOCLk/build/release/bin/postgresql.jar:/tmp/dist-test-taskhWOCLk/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.14.147.212:59701/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-98mk.c.gcp-upstream.internal';"
2026-09-29 22:51:19,334 [I] CORE_DB_SCHEMA import status has been updated
2026-09-29 22:51:19,334 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskhWOCLk/build/release/bin/postgresql.jar:/tmp/dist-test-taskhWOCLk/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.14.147.212:59701/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-09-29 22:51:19,569 [I] DB_PATCHES have already been applied
I20260929 22:51:19.574705 14927 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790722213368738-14927-0/minicluster-data/ranger-admin:/tmp/dist-test-taskhWOCLk/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskhWOCLk/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskhWOCLk/build/release/bin/postgresql.jar:/tmp/dist-test-taskhWOCLk/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260929 22:51:19.574779 14927 mini_ranger.cc:193] Using host: 127.14.147.212
I20260929 22:51:19.577713 14927 mini_ranger.cc:227] Ranger admin URL: http://127.14.147.212:59043
Sep 29, 2026 10:51:19 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 29, 2026 10:51:19 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 29, 2026 10:51:20 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790722213368738-14927-0/minicluster-data/ranger-admin/ews/webapp
Sep 29, 2026 10:51:20 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790722213368738-14927-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Sep 29, 2026 10:51:20 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790722213368738-14927-0/minicluster-data/ranger-admin/ews/webapp] .....
Sep 29, 2026 10:51:20 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790722213368738-14927-0/minicluster-data/ranger-admin/ews/webapp].
Sep 29, 2026 10:51:20 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260929 22:51:21.620180 14927 mini_ranger.cc:160] Time spent starting Ranger: real 9.640s user 0.001s sys 0.010s
Sep 29, 2026 10:51:41 PM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Sep 29, 2026 10:51:41 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.UserREST
Sep 29, 2026 10:51:41 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Sep 29, 2026 10:51:42 PM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Sep 29, 2026 10:51:42 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Sep 29, 2026 10:51:42 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Sep 29, 2026 10:51:42 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Sep 29, 2026 10:51:42 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Sep 29, 2026 10:51:42 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Sep 29, 2026 10:51:42 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Sep 29, 2026 10:51:42 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Sep 29, 2026 10:51:42 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Sep 29, 2026 10:51:42 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Sep 29, 2026 10:51:42 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Sep 29, 2026 10:51:42 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Sep 29, 2026 10:51:42 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Sep 29, 2026 10:51:42 PM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Sep 29, 2026 10:51:42 PM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260929 22:51:43.538606 14927 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.14.147.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.14.147.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.14.147.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790722213368738-14927-0/krb5kdc/rangerkms_127.14.147.212@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.14.147.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790722213368738-14927-0/krb5kdc/rangerkms_127.14.147.212@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.14.147.212@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.14.147.212@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.14.147.212@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790722213368738-14927-0/krb5kdc/HTTP_127.14.147.212@KRBTEST.COM.keytab.
Entry for principal HTTP/127.14.147.212@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790722213368738-14927-0/krb5kdc/HTTP_127.14.147.212@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Sep 29 22:51:43 dist-test-slave-98mk krb5kdc[25677](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790722303, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260929 22:51:43.626706 14927 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260929 22:51:43.636085 14927 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260929 22:51:44.517858 14927 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260929 22:51:44.517961 14927 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790722213368738-14927-0/minicluster-data/ranger-kms
I20260929 22:51:44.517980 14927 mini_ranger_kms.cc:227] Using postgres at 127.14.147.212:59701
2026-09-29 22:51:44,781 [I] DB FLAVOR :POSTGRES
2026-09-29 22:51:44,782 [I] --------- Verifying Ranger DB connection ---------
2026-09-29 22:51:44,782 [I] Checking connection
2026-09-29 22:51:44,782 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskhWOCLk/build/release/bin/postgresql.jar:/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790722213368738-14927-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.14.147.212:59701/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-09-29 22:51:45,021 [I] connection success
2026-09-29 22:51:45,021 [I] --------- Verifying Ranger DB tables ---------
2026-09-29 22:51:45,022 [I] Verifying table ranger_masterkey in database rangerkms
2026-09-29 22:51:45,022 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskhWOCLk/build/release/bin/postgresql.jar:/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790722213368738-14927-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.14.147.212:59701/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-09-29 22:51:45,268 [I] Table ranger_masterkey does not exist in database rangerkms
2026-09-29 22:51:45,268 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-29 22:51:45,268 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-09-29 22:51:45,268 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskhWOCLk/build/release/bin/postgresql.jar:/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790722213368738-14927-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.14.147.212:59701/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790722213368738-14927-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-09-29 22:51:45,530 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260929 22:51:46.009222 14927 mini_ranger_kms.cc:310] Created kms service
I20260929 22:51:46.230391 14927 mini_ranger_kms.cc:326] Created kudu user
I20260929 22:51:46.277840 14927 mini_ranger_kms.cc:343] Created rangerkms user
Sep 29 22:51:46 dist-test-slave-98mk krb5kdc[25677](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790722303, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.14.147.212@KRBTEST.COM
I20260929 22:51:46.714919 14927 mini_ranger_kms.cc:384] Added ranger policy
I20260929 22:51:46.715056 14927 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790722213368738-14927-0/minicluster-data/ranger-kms:/tmp/dist-test-taskhWOCLk/build/release/bin/postgresql.jar:/tmp/dist-test-taskhWOCLk/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskhWOCLk/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260929 22:51:46.715076 14927 mini_ranger_kms.cc:240] Using host: 127.14.147.212
I20260929 22:51:46.719228 14927 mini_ranger_kms.cc:276] Ranger KMS PID: 26075
I20260929 22:51:46.719375 14927 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.14.147.212:37345
Sep 29, 2026 10:51:47 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 29, 2026 10:51:47 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 29, 2026 10:51:47 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790722213368738-14927-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Sep 29, 2026 10:51:47 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790722213368738-14927-0/minicluster-data/ranger-kms/ews/webapp] .....
Sep 29, 2026 10:51:48 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790722213368738-14927-0/minicluster-data/ranger-kms/ews/webapp].
Sep 29, 2026 10:51:48 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260929 22:51:48.775923 14927 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.149s user 0.007s sys 0.188s
I20260929 22:51:48.776034 14927 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260929 22:51:48.776119 14927 mini_ranger_kms.cc:401] 127.14.147.212:37345/kms/v1/keys
Sep 29 22:51:48 dist-test-slave-98mk krb5kdc[25677](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790722308, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.14.147.212@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Sep 29 22:51:49 dist-test-slave-98mk krb5kdc[25677](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790722308, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.14.147.212@KRBTEST.COM for HTTP/127.14.147.212@KRBTEST.COM
Sep 29 22:51:49 dist-test-slave-98mk krb5kdc[25677](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790722308, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.14.147.212@KRBTEST.COM for HTTP/127.14.147.212@KRBTEST.COM
Sep 29 22:51:51 dist-test-slave-98mk krb5kdc[25677](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790722311, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.14.147.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.147.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.147.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790722213368738-14927-0/krb5kdc/kudu_127.14.147.254.keytab.
Entry for principal kudu/127.14.147.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790722213368738-14927-0/krb5kdc/kudu_127.14.147.254.keytab.
I20260929 22:51:52.000157 14927 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhWOCLk/build/release/bin/kudu
/tmp/dist-test-taskhWOCLk/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790722213368738-14927-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790722213368738-14927-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790722213368738-14927-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790722213368738-14927-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.147.254:34001
--webserver_interface=127.14.147.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790722213368738-14927-0/krb5kdc/kudu_127.14.147.254.keytab
--principal=kudu/127.14.147.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.147.212:39003
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.147.254:34001
--ranger_config_path=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790722213368738-14927-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.14.147.212:37345/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790722213368738-14927-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790722213368738-14927-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790722213368738-14927-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 22:51:52.084386 26125 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 22:51:52.084632 26125 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 22:51:52.084654 26125 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 22:51:52.085911 26125 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 22:51:52.085964 26125 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 22:51:52.085978 26125 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 22:51:52.085988 26125 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 22:51:52.086000 26125 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 22:51:52.088039 26125 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.147.212:39003
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790722213368738-14927-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790722213368738-14927-0/minicluster-data/master-0/wal
--ranger_kms_url=127.14.147.212:37345/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.14.147.254:34001
--ranger_config_path=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790722213368738-14927-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790722213368738-14927-0/krb5kdc/kudu_127.14.147.254.keytab
--principal=kudu/127.14.147.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.147.254:34001
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790722213368738-14927-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.14.147.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26125
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790722213368738-14927-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 21:58:54 UTC on ccycloud.quasar-xguoti.root.comops.site
build id 4681390
I20260929 22:51:52.088286 26125 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 22:51:52.088518 26125 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 22:51:52.093356 26131 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 22:51:52.093338 26133 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 22:51:52.093390 26130 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 22:51:52.094267 26125 server_base.cc:1034] running on GCE node
I20260929 22:51:52.094544 26125 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 22:51:52.094825 26125 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 22:51:52.095976 26125 hybrid_clock.cc:648] HybridClock initialized: now 1790722312095959 us; error 31 us; skew 500 ppm
Sep 29 22:51:52 dist-test-slave-98mk krb5kdc[25677](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790722312, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.147.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 22:51:52.098461 26125 init.cc:377] Logged in from keytab as kudu/127.14.147.254@KRBTEST.COM (short username kudu)
I20260929 22:51:52.098933 26125 webserver.cc:460] Webserver started at http://127.14.147.254:45473/ using document root <none> and password file <none>
I20260929 22:51:52.099134 26125 fs_manager.cc:359] Metadata directory not provided
I20260929 22:51:52.099174 26125 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 22:51:52.099295 26125 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 29 22:51:52 dist-test-slave-98mk krb5kdc[25677](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790722312, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.147.254@KRBTEST.COM for HTTP/127.14.147.212@KRBTEST.COM
I20260929 22:51:52.152518 26125 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790722213368738-14927-0/minicluster-data/master-0/data/instance:
uuid: "fd0f765e325e4140bf9771ccb19f3e4a"
format_stamp: "Formatted at 2026-09-29 22:51:52 on dist-test-slave-98mk"
server_key: "a46c1007e7393268371b55ede99bcede"
server_key_iv: "3ca28f3c1388bafb6e87a0e36f5bcbcf"
server_key_version: "kuduclusterkey@0"
I20260929 22:51:52.153008 26125 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790722213368738-14927-0/minicluster-data/master-0/wal/instance:
uuid: "fd0f765e325e4140bf9771ccb19f3e4a"
format_stamp: "Formatted at 2026-09-29 22:51:52 on dist-test-slave-98mk"
server_key: "a46c1007e7393268371b55ede99bcede"
server_key_iv: "3ca28f3c1388bafb6e87a0e36f5bcbcf"
server_key_version: "kuduclusterkey@0"
I20260929 22:51:52.154421 26125 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260929 22:51:52.170040 26140 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 22:51:52.170233 26125 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260929 22:51:52.170336 26125 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790722213368738-14927-0/minicluster-data/master-0/data,/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790722213368738-14927-0/minicluster-data/master-0/wal
uuid: "fd0f765e325e4140bf9771ccb19f3e4a"
format_stamp: "Formatted at 2026-09-29 22:51:52 on dist-test-slave-98mk"
server_key: "a46c1007e7393268371b55ede99bcede"
server_key_iv: "3ca28f3c1388bafb6e87a0e36f5bcbcf"
server_key_version: "kuduclusterkey@0"
I20260929 22:51:52.170462 26125 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790722213368738-14927-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790722213368738-14927-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790722213368738-14927-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 22:51:52.186069 26125 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 22:51:52.187942 26125 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 22:51:52.188076 26125 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 22:51:52.192612 26125 rpc_server.cc:307] RPC server started. Bound to: 127.14.147.254:34001
I20260929 22:51:52.192629 26193 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.147.254:34001 every 8 connection(s)
I20260929 22:51:52.193008 26125 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790722213368738-14927-0/minicluster-data/master-0/data/info.pb
I20260929 22:51:52.193650 26194 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 22:51:52.196022 26194 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fd0f765e325e4140bf9771ccb19f3e4a: Bootstrap starting.
I20260929 22:51:52.196169 14927 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhWOCLk/build/release/bin/kudu as pid 26125
I20260929 22:51:52.196281 14927 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790722213368738-14927-0/minicluster-data/master-0/wal/instance
I20260929 22:51:52.196455 14927 external_mini_cluster.cc:1428] Setting key 8e463a2dcd1318421d317fc7c3b1e4f4
I20260929 22:51:52.196790 26194 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P fd0f765e325e4140bf9771ccb19f3e4a: Neither blocks nor log segments found. Creating new log.
I20260929 22:51:52.197803 26194 log.cc:824] T 00000000000000000000000000000000 P fd0f765e325e4140bf9771ccb19f3e4a: Log is configured to *not* fsync() on all Append() calls
I20260929 22:51:52.198571 26194 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fd0f765e325e4140bf9771ccb19f3e4a: No bootstrap required, opened a new log
I20260929 22:51:52.200220 26194 raft_consensus.cc:348] T 00000000000000000000000000000000 P fd0f765e325e4140bf9771ccb19f3e4a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fd0f765e325e4140bf9771ccb19f3e4a" member_type: VOTER last_known_addr { host: "127.14.147.254" port: 34001 } }
I20260929 22:51:52.200366 26194 raft_consensus.cc:374] T 00000000000000000000000000000000 P fd0f765e325e4140bf9771ccb19f3e4a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 22:51:52.200399 26194 raft_consensus.cc:729] T 00000000000000000000000000000000 P fd0f765e325e4140bf9771ccb19f3e4a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fd0f765e325e4140bf9771ccb19f3e4a, State: Initialized, Role: FOLLOWER
I20260929 22:51:52.200565 26194 consensus_queue.cc:260] T 00000000000000000000000000000000 P fd0f765e325e4140bf9771ccb19f3e4a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fd0f765e325e4140bf9771ccb19f3e4a" member_type: VOTER last_known_addr { host: "127.14.147.254" port: 34001 } }
I20260929 22:51:52.200654 26194 raft_consensus.cc:388] T 00000000000000000000000000000000 P fd0f765e325e4140bf9771ccb19f3e4a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 22:51:52.200686 26194 raft_consensus.cc:482] T 00000000000000000000000000000000 P fd0f765e325e4140bf9771ccb19f3e4a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 22:51:52.200747 26194 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fd0f765e325e4140bf9771ccb19f3e4a [term 0 FOLLOWER]: Advancing to term 1
I20260929 22:51:52.201340 26194 raft_consensus.cc:504] T 00000000000000000000000000000000 P fd0f765e325e4140bf9771ccb19f3e4a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fd0f765e325e4140bf9771ccb19f3e4a" member_type: VOTER last_known_addr { host: "127.14.147.254" port: 34001 } }
I20260929 22:51:52.201453 26194 leader_election.cc:302] T 00000000000000000000000000000000 P fd0f765e325e4140bf9771ccb19f3e4a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: fd0f765e325e4140bf9771ccb19f3e4a; no voters:
I20260929 22:51:52.201685 26194 leader_election.cc:288] T 00000000000000000000000000000000 P fd0f765e325e4140bf9771ccb19f3e4a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 22:51:52.201740 26199 raft_consensus.cc:2781] T 00000000000000000000000000000000 P fd0f765e325e4140bf9771ccb19f3e4a [term 1 FOLLOWER]: Leader election won for term 1
I20260929 22:51:52.201969 26199 raft_consensus.cc:686] T 00000000000000000000000000000000 P fd0f765e325e4140bf9771ccb19f3e4a [term 1 LEADER]: Becoming Leader. State: Replica: fd0f765e325e4140bf9771ccb19f3e4a, State: Running, Role: LEADER
I20260929 22:51:52.202030 26194 sys_catalog.cc:564] T 00000000000000000000000000000000 P fd0f765e325e4140bf9771ccb19f3e4a [sys.catalog]: configured and running, proceeding with master startup.
I20260929 22:51:52.202145 26199 consensus_queue.cc:237] T 00000000000000000000000000000000 P fd0f765e325e4140bf9771ccb19f3e4a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fd0f765e325e4140bf9771ccb19f3e4a" member_type: VOTER last_known_addr { host: "127.14.147.254" port: 34001 } }
I20260929 22:51:52.203804 26194 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790722213368738-14927-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260929 22:51:52.204365 26200 sys_catalog.cc:455] T 00000000000000000000000000000000 P fd0f765e325e4140bf9771ccb19f3e4a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "fd0f765e325e4140bf9771ccb19f3e4a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fd0f765e325e4140bf9771ccb19f3e4a" member_type: VOTER last_known_addr { host: "127.14.147.254" port: 34001 } } }
I20260929 22:51:52.204453 26200 sys_catalog.cc:458] T 00000000000000000000000000000000 P fd0f765e325e4140bf9771ccb19f3e4a [sys.catalog]: This master's current role is: LEADER
I20260929 22:51:52.204624 26201 sys_catalog.cc:455] T 00000000000000000000000000000000 P fd0f765e325e4140bf9771ccb19f3e4a [sys.catalog]: SysCatalogTable state changed. Reason: New leader fd0f765e325e4140bf9771ccb19f3e4a. Latest consensus state: current_term: 1 leader_uuid: "fd0f765e325e4140bf9771ccb19f3e4a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fd0f765e325e4140bf9771ccb19f3e4a" member_type: VOTER last_known_addr { host: "127.14.147.254" port: 34001 } } }
I20260929 22:51:52.204758 26201 sys_catalog.cc:458] T 00000000000000000000000000000000 P fd0f765e325e4140bf9771ccb19f3e4a [sys.catalog]: This master's current role is: LEADER
Sep 29 22:51:52 dist-test-slave-98mk krb5kdc[25677](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790722311, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.147.254@KRBTEST.COM
I20260929 22:51:52.206766 26215 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 22:51:52.207793 26215 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 22:51:52.211864 26197 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:51:52.197236 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47434 (local address 127.14.147.254:34001)
0929 22:51:52.197554 (+ 318us) server_negotiation.cc:206] Beginning negotiation
0929 22:51:52.197564 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 22:51:52.197601 (+ 37us) server_negotiation.cc:407] Connection header received
0929 22:51:52.198238 (+ 637us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 22:51:52.198248 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 22:51:52.198314 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 22:51:52.198461 (+ 147us) server_negotiation.cc:226] Negotiated authn=SASL
0929 22:51:52.199231 (+ 770us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:51:52.201198 (+ 1967us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:51:52.202264 (+ 1066us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:51:52.202894 (+ 630us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:51:52.203009 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 22:51:52.206251 (+ 3242us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 22:51:52.206286 (+ 35us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 22:51:52.206286 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 22:51:52.206366 (+ 80us) server_negotiation.cc:922] calling sasl_server_start()
0929 22:51:52.209431 (+ 3065us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 22:51:52.209854 (+ 423us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 22:51:52.209858 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 22:51:52.209858 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 22:51:52.209983 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 22:51:52.210218 (+ 235us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 22:51:52.210219 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 22:51:52.210219 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 22:51:52.210682 (+ 463us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 22:51:52.211218 (+ 536us) server_negotiation.cc:1035] Waiting for connection context
0929 22:51:52.211290 (+ 72us) server_negotiation.cc:299] Negotiation successful
0929 22:51:52.211581 (+ 291us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":252,"thread_start_us":180,"threads_started":1}
I20260929 22:51:52.213187 26215 catalog_manager.cc:1347] Generated new cluster ID: 04fa464e178046a29b3319be115ed732
I20260929 22:51:52.213275 26215 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 22:51:52.235518 26215 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 22:51:52.236395 26215 catalog_manager.cc:1504] Loading token signing keys...
I20260929 22:51:52.242507 26215 catalog_manager.cc:5868] T 00000000000000000000000000000000 P fd0f765e325e4140bf9771ccb19f3e4a: Generated new TSK 0
I20260929 22:51:52.242837 26215 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.14.147.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.147.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.147.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790722213368738-14927-0/krb5kdc/kudu_127.14.147.193.keytab.
Entry for principal kudu/127.14.147.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790722213368738-14927-0/krb5kdc/kudu_127.14.147.193.keytab.
I20260929 22:51:52.974457 14927 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhWOCLk/build/release/bin/kudu
/tmp/dist-test-taskhWOCLk/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790722213368738-14927-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790722213368738-14927-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790722213368738-14927-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790722213368738-14927-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.14.147.212:37345/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.147.193:0
--local_ip_for_outbound_sockets=127.14.147.193
--webserver_interface=127.14.147.193
--webserver_port=0
--tserver_master_addrs=127.14.147.254:34001
--keytab_file=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790722213368738-14927-0/krb5kdc/kudu_127.14.147.193.keytab
--principal=kudu/127.14.147.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.147.212:39003
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790722213368738-14927-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790722213368738-14927-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790722213368738-14927-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 22:51:53.095564 26239 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 22:51:53.095790 26239 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 22:51:53.095815 26239 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 22:51:53.097091 26239 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 22:51:53.097142 26239 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 22:51:53.097169 26239 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.147.193
I20260929 22:51:53.098781 26239 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.147.212:39003
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790722213368738-14927-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790722213368738-14927-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.14.147.212:37345/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790722213368738-14927-0/krb5kdc/kudu_127.14.147.193.keytab
--principal=kudu/127.14.147.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.147.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790722213368738-14927-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.14.147.193
--webserver_port=0
--tserver_master_addrs=127.14.147.254:34001
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26239
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.147.193
--log_dir=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790722213368738-14927-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 21:58:54 UTC on ccycloud.quasar-xguoti.root.comops.site
build id 4681390
I20260929 22:51:53.099031 26239 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 22:51:53.099261 26239 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 22:51:53.105290 26244 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 22:51:53.105288 26245 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 22:51:53.105649 26247 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 22:51:53.106099 26239 server_base.cc:1034] running on GCE node
I20260929 22:51:53.106333 26239 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 22:51:53.106622 26239 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 22:51:53.114009 26239 hybrid_clock.cc:648] HybridClock initialized: now 1790722313113979 us; error 34 us; skew 500 ppm
Sep 29 22:51:53 dist-test-slave-98mk krb5kdc[25677](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790722313, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.147.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 22:51:53.117403 26239 init.cc:377] Logged in from keytab as kudu/127.14.147.193@KRBTEST.COM (short username kudu)
I20260929 22:51:53.117980 26239 webserver.cc:460] Webserver started at http://127.14.147.193:43219/ using document root <none> and password file <none>
I20260929 22:51:53.118238 26239 fs_manager.cc:359] Metadata directory not provided
I20260929 22:51:53.118299 26239 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 22:51:53.118429 26239 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 29 22:51:53 dist-test-slave-98mk krb5kdc[25677](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790722313, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.147.193@KRBTEST.COM for HTTP/127.14.147.212@KRBTEST.COM
I20260929 22:51:53.134538 26239 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790722213368738-14927-0/minicluster-data/ts-0/data/instance:
uuid: "78ec93dc79b041ddb17d95d6241351bd"
format_stamp: "Formatted at 2026-09-29 22:51:53 on dist-test-slave-98mk"
server_key: "194edd385fa531cf4fc9405a950d5308"
server_key_iv: "bfec5d2c7558c29d18ba331b823ba551"
server_key_version: "kuduclusterkey@0"
I20260929 22:51:53.135061 26239 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790722213368738-14927-0/minicluster-data/ts-0/wal/instance:
uuid: "78ec93dc79b041ddb17d95d6241351bd"
format_stamp: "Formatted at 2026-09-29 22:51:53 on dist-test-slave-98mk"
server_key: "194edd385fa531cf4fc9405a950d5308"
server_key_iv: "bfec5d2c7558c29d18ba331b823ba551"
server_key_version: "kuduclusterkey@0"
I20260929 22:51:53.137004 26239 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.001s sys 0.000s
I20260929 22:51:53.150478 26254 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 22:51:53.150717 26239 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260929 22:51:53.150808 26239 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790722213368738-14927-0/minicluster-data/ts-0/data,/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790722213368738-14927-0/minicluster-data/ts-0/wal
uuid: "78ec93dc79b041ddb17d95d6241351bd"
format_stamp: "Formatted at 2026-09-29 22:51:53 on dist-test-slave-98mk"
server_key: "194edd385fa531cf4fc9405a950d5308"
server_key_iv: "bfec5d2c7558c29d18ba331b823ba551"
server_key_version: "kuduclusterkey@0"
I20260929 22:51:53.150903 26239 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790722213368738-14927-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790722213368738-14927-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790722213368738-14927-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 22:51:53.164652 26239 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 22:51:53.166960 26239 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 22:51:53.167148 26239 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 22:51:53.167482 26239 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 22:51:53.167891 26239 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 22:51:53.167990 26239 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 22:51:53.168081 26239 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 22:51:53.168149 26239 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 22:51:53.174268 26239 rpc_server.cc:307] RPC server started. Bound to: 127.14.147.193:44171
I20260929 22:51:53.174731 26239 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790722213368738-14927-0/minicluster-data/ts-0/data/info.pb
I20260929 22:51:53.180116 26369 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.147.193:44171 every 8 connection(s)
I20260929 22:51:53.182929 14927 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhWOCLk/build/release/bin/kudu as pid 26239
I20260929 22:51:53.183033 14927 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790722213368738-14927-0/minicluster-data/ts-0/wal/instance
I20260929 22:51:53.183236 14927 external_mini_cluster.cc:1428] Setting key 3364f712758f1be565e36a70bf277922
Sep 29 22:51:53 dist-test-slave-98mk krb5kdc[25677](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790722313, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.147.193@KRBTEST.COM for kudu/127.14.147.254@KRBTEST.COM
I20260929 22:51:53.191679 26373 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:51:53.180955 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.147.193:52827 (local address 127.14.147.254:34001)
0929 22:51:53.182052 (+ 1097us) server_negotiation.cc:206] Beginning negotiation
0929 22:51:53.182054 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 22:51:53.182062 (+ 8us) server_negotiation.cc:407] Connection header received
0929 22:51:53.182088 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 22:51:53.182088 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 22:51:53.182101 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 22:51:53.182166 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0929 22:51:53.183006 (+ 840us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:51:53.184273 (+ 1267us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:51:53.185251 (+ 978us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:51:53.185556 (+ 305us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:51:53.185623 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 22:51:53.188275 (+ 2652us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 22:51:53.188276 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 22:51:53.188277 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 22:51:53.188284 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0929 22:51:53.189993 (+ 1709us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 22:51:53.190508 (+ 515us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 22:51:53.190509 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 22:51:53.190509 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 22:51:53.190572 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 22:51:53.190822 (+ 250us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 22:51:53.190823 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 22:51:53.190824 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 22:51:53.190972 (+ 148us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 22:51:53.191026 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0929 22:51:53.191485 (+ 459us) server_negotiation.cc:299] Negotiation successful
0929 22:51:53.191575 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1066,"thread_start_us":75,"threads_started":1}
I20260929 22:51:53.194864 26372 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:51:53.180883 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.147.254:34001 (local address 127.14.147.193:52827)
0929 22:51:53.181164 (+ 281us) negotiation.cc:107] Waiting for socket to connect
0929 22:51:53.181211 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0929 22:51:53.181855 (+ 644us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 22:51:53.182304 (+ 449us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 22:51:53.182309 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 22:51:53.182595 (+ 286us) client_negotiation.cc:189] Negotiated authn=SASL
0929 22:51:53.182889 (+ 294us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 22:51:53.182895 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 22:51:53.184365 (+ 1470us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 22:51:53.184366 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 22:51:53.185165 (+ 799us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 22:51:53.185167 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 22:51:53.185696 (+ 529us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 22:51:53.185697 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 22:51:53.185850 (+ 153us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 22:51:53.186457 (+ 607us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 22:51:53.186467 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0929 22:51:53.188161 (+ 1694us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 22:51:53.190085 (+ 1924us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 22:51:53.190092 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 22:51:53.190094 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 22:51:53.190424 (+ 330us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 22:51:53.190633 (+ 209us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 22:51:53.190634 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 22:51:53.190634 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 22:51:53.190761 (+ 127us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 22:51:53.191051 (+ 290us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 22:51:53.191057 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 22:51:53.191224 (+ 167us) client_negotiation.cc:769] Sending connection context
0929 22:51:53.191327 (+ 103us) client_negotiation.cc:240] Negotiation successful
0929 22:51:53.192154 (+ 827us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":237,"thread_start_us":95,"threads_started":1}
I20260929 22:51:53.195940 26370 heartbeater.cc:347] Connected to a master server at 127.14.147.254:34001
I20260929 22:51:53.196040 26370 heartbeater.cc:464] Registering TS with master...
I20260929 22:51:53.196229 26370 heartbeater.cc:511] Master 127.14.147.254:34001 requested a full tablet report, sending...
I20260929 22:51:53.196769 26158 ts_manager.cc:194] Registered new tserver with Master: 78ec93dc79b041ddb17d95d6241351bd (127.14.147.193:44171)
I20260929 22:51:53.197642 26158 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.147.193@KRBTEST.COM'} at 127.14.147.193:52827
WARNING: no policy specified for kudu/127.14.147.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.147.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.147.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790722213368738-14927-0/krb5kdc/kudu_127.14.147.194.keytab.
Entry for principal kudu/127.14.147.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790722213368738-14927-0/krb5kdc/kudu_127.14.147.194.keytab.
I20260929 22:51:53.220263 14927 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhWOCLk/build/release/bin/kudu
/tmp/dist-test-taskhWOCLk/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790722213368738-14927-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790722213368738-14927-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790722213368738-14927-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790722213368738-14927-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.14.147.212:37345/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.147.194:0
--local_ip_for_outbound_sockets=127.14.147.194
--webserver_interface=127.14.147.194
--webserver_port=0
--tserver_master_addrs=127.14.147.254:34001
--keytab_file=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790722213368738-14927-0/krb5kdc/kudu_127.14.147.194.keytab
--principal=kudu/127.14.147.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.147.212:39003
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790722213368738-14927-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790722213368738-14927-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790722213368738-14927-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 29 22:51:53 dist-test-slave-98mk krb5kdc[25677](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790722313, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.147.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260929 22:51:53.336841 26376 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 22:51:53.337155 26376 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 22:51:53.337193 26376 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 22:51:53.339113 26376 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 22:51:53.339183 26376 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 22:51:53.339223 26376 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.147.194
I20260929 22:51:53.341747 26376 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.147.212:39003
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790722213368738-14927-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790722213368738-14927-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.14.147.212:37345/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790722213368738-14927-0/krb5kdc/kudu_127.14.147.194.keytab
--principal=kudu/127.14.147.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.147.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790722213368738-14927-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.14.147.194
--webserver_port=0
--tserver_master_addrs=127.14.147.254:34001
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26376
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.147.194
--log_dir=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790722213368738-14927-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 21:58:54 UTC on ccycloud.quasar-xguoti.root.comops.site
build id 4681390
I20260929 22:51:53.342041 26376 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 22:51:53.342379 26376 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 22:51:53.348948 26384 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 22:51:53.349357 26383 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 22:51:53.349481 26386 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 22:51:53.350639 26376 server_base.cc:1034] running on GCE node
I20260929 22:51:53.350854 26376 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 22:51:53.351114 26376 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 22:51:53.352294 26376 hybrid_clock.cc:648] HybridClock initialized: now 1790722313352265 us; error 38 us; skew 500 ppm
Sep 29 22:51:53 dist-test-slave-98mk krb5kdc[25677](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790722313, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.147.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 22:51:53.355753 26376 init.cc:377] Logged in from keytab as kudu/127.14.147.194@KRBTEST.COM (short username kudu)
I20260929 22:51:53.356269 26376 webserver.cc:460] Webserver started at http://127.14.147.194:41889/ using document root <none> and password file <none>
I20260929 22:51:53.356531 26376 fs_manager.cc:359] Metadata directory not provided
I20260929 22:51:53.356602 26376 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 22:51:53.356738 26376 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 29 22:51:53 dist-test-slave-98mk krb5kdc[25677](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790722313, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.147.194@KRBTEST.COM for HTTP/127.14.147.212@KRBTEST.COM
I20260929 22:51:53.380714 26376 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790722213368738-14927-0/minicluster-data/ts-1/data/instance:
uuid: "adb47a32e0a3483ab2b65bccba6af58e"
format_stamp: "Formatted at 2026-09-29 22:51:53 on dist-test-slave-98mk"
server_key: "295517ed721fa1fc123465457113b42b"
server_key_iv: "895c1caf16a9c35771ea5d1c9693c04a"
server_key_version: "kuduclusterkey@0"
I20260929 22:51:53.381143 26376 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790722213368738-14927-0/minicluster-data/ts-1/wal/instance:
uuid: "adb47a32e0a3483ab2b65bccba6af58e"
format_stamp: "Formatted at 2026-09-29 22:51:53 on dist-test-slave-98mk"
server_key: "295517ed721fa1fc123465457113b42b"
server_key_iv: "895c1caf16a9c35771ea5d1c9693c04a"
server_key_version: "kuduclusterkey@0"
I20260929 22:51:53.383838 26376 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.003s sys 0.000s
I20260929 22:51:53.409461 26393 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 22:51:53.409787 26376 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260929 22:51:53.409889 26376 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790722213368738-14927-0/minicluster-data/ts-1/data,/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790722213368738-14927-0/minicluster-data/ts-1/wal
uuid: "adb47a32e0a3483ab2b65bccba6af58e"
format_stamp: "Formatted at 2026-09-29 22:51:53 on dist-test-slave-98mk"
server_key: "295517ed721fa1fc123465457113b42b"
server_key_iv: "895c1caf16a9c35771ea5d1c9693c04a"
server_key_version: "kuduclusterkey@0"
I20260929 22:51:53.409992 26376 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790722213368738-14927-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790722213368738-14927-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790722213368738-14927-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 22:51:53.437440 26376 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 22:51:53.440119 26376 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 22:51:53.440333 26376 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 22:51:53.440610 26376 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 22:51:53.441037 26376 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 22:51:53.441125 26376 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 22:51:53.441201 26376 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 22:51:53.441256 26376 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 22:51:53.448482 26376 rpc_server.cc:307] RPC server started. Bound to: 127.14.147.194:32833
I20260929 22:51:53.449026 26376 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790722213368738-14927-0/minicluster-data/ts-1/data/info.pb
I20260929 22:51:53.450250 14927 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhWOCLk/build/release/bin/kudu as pid 26376
I20260929 22:51:53.450348 14927 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790722213368738-14927-0/minicluster-data/ts-1/wal/instance
I20260929 22:51:53.450567 14927 external_mini_cluster.cc:1428] Setting key 037f3dc758358bd6381e4f6f5b399e01
I20260929 22:51:53.451964 26508 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.147.194:32833 every 8 connection(s)
Sep 29 22:51:53 dist-test-slave-98mk krb5kdc[25677](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790722313, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.147.194@KRBTEST.COM for kudu/127.14.147.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.14.147.195@KRBTEST.COM; defaulting to no policy
I20260929 22:51:53.464437 26373 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:51:53.453426 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.147.194:41343 (local address 127.14.147.254:34001)
0929 22:51:53.453474 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0929 22:51:53.453476 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 22:51:53.453823 (+ 347us) server_negotiation.cc:407] Connection header received
0929 22:51:53.454296 (+ 473us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 22:51:53.454298 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 22:51:53.454313 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 22:51:53.454361 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0929 22:51:53.455188 (+ 827us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:51:53.455820 (+ 632us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:51:53.456775 (+ 955us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:51:53.457050 (+ 275us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:51:53.457131 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 22:51:53.459691 (+ 2560us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 22:51:53.459692 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 22:51:53.459693 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 22:51:53.459700 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0929 22:51:53.461294 (+ 1594us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 22:51:53.462931 (+ 1637us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 22:51:53.462933 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 22:51:53.462933 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 22:51:53.463002 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 22:51:53.463499 (+ 497us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 22:51:53.463500 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 22:51:53.463500 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 22:51:53.463664 (+ 164us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 22:51:53.463710 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0929 22:51:53.464183 (+ 473us) server_negotiation.cc:299] Negotiation successful
0929 22:51:53.464291 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.147.195@KRBTEST.COM" created.
I20260929 22:51:53.467238 26512 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:51:53.453148 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.147.254:34001 (local address 127.14.147.194:41343)
0929 22:51:53.453704 (+ 556us) negotiation.cc:107] Waiting for socket to connect
0929 22:51:53.453738 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0929 22:51:53.454208 (+ 470us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 22:51:53.454492 (+ 284us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 22:51:53.454496 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 22:51:53.454744 (+ 248us) client_negotiation.cc:189] Negotiated authn=SASL
0929 22:51:53.455069 (+ 325us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 22:51:53.455075 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 22:51:53.455911 (+ 836us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 22:51:53.455912 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 22:51:53.456695 (+ 783us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 22:51:53.456697 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 22:51:53.457199 (+ 502us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 22:51:53.457200 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 22:51:53.457344 (+ 144us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 22:51:53.457868 (+ 524us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 22:51:53.457878 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0929 22:51:53.459582 (+ 1704us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 22:51:53.462456 (+ 2874us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 22:51:53.462467 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 22:51:53.462470 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 22:51:53.462804 (+ 334us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 22:51:53.463206 (+ 402us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 22:51:53.463207 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 22:51:53.463207 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 22:51:53.463368 (+ 161us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 22:51:53.463759 (+ 391us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 22:51:53.463767 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 22:51:53.464019 (+ 252us) client_negotiation.cc:769] Sending connection context
0929 22:51:53.466730 (+ 2711us) client_negotiation.cc:240] Negotiation successful
0929 22:51:53.466964 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":514,"thread_start_us":89,"threads_started":1}
I20260929 22:51:53.468011 26509 heartbeater.cc:347] Connected to a master server at 127.14.147.254:34001
I20260929 22:51:53.468179 26509 heartbeater.cc:464] Registering TS with master...
I20260929 22:51:53.468441 26509 heartbeater.cc:511] Master 127.14.147.254:34001 requested a full tablet report, sending...
I20260929 22:51:53.468879 26158 ts_manager.cc:194] Registered new tserver with Master: adb47a32e0a3483ab2b65bccba6af58e (127.14.147.194:32833)
I20260929 22:51:53.469412 26158 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.147.194@KRBTEST.COM'} at 127.14.147.194:41343
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.147.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790722213368738-14927-0/krb5kdc/kudu_127.14.147.195.keytab.
Entry for principal kudu/127.14.147.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790722213368738-14927-0/krb5kdc/kudu_127.14.147.195.keytab.
I20260929 22:51:53.486246 14927 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhWOCLk/build/release/bin/kudu
/tmp/dist-test-taskhWOCLk/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790722213368738-14927-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790722213368738-14927-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790722213368738-14927-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790722213368738-14927-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.14.147.212:37345/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.147.195:0
--local_ip_for_outbound_sockets=127.14.147.195
--webserver_interface=127.14.147.195
--webserver_port=0
--tserver_master_addrs=127.14.147.254:34001
--keytab_file=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790722213368738-14927-0/krb5kdc/kudu_127.14.147.195.keytab
--principal=kudu/127.14.147.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.147.212:39003
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790722213368738-14927-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790722213368738-14927-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790722213368738-14927-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 22:51:53.612515 26514 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 22:51:53.612886 26514 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 22:51:53.612973 26514 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 22:51:53.614820 26514 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 22:51:53.615058 26514 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 22:51:53.615162 26514 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.147.195
I20260929 22:51:53.618698 26514 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.147.212:39003
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790722213368738-14927-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790722213368738-14927-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.14.147.212:37345/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790722213368738-14927-0/krb5kdc/kudu_127.14.147.195.keytab
--principal=kudu/127.14.147.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.147.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790722213368738-14927-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.14.147.195
--webserver_port=0
--tserver_master_addrs=127.14.147.254:34001
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26514
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.147.195
--log_dir=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790722213368738-14927-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 21:58:54 UTC on ccycloud.quasar-xguoti.root.comops.site
build id 4681390
I20260929 22:51:53.619216 26514 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 22:51:53.619571 26514 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 22:51:53.626067 26520 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 22:51:53.626889 26522 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 22:51:53.627185 26514 server_base.cc:1034] running on GCE node
W20260929 22:51:53.627856 26519 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 22:51:53.628156 26514 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 22:51:53.628427 26514 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 22:51:53.629585 26514 hybrid_clock.cc:648] HybridClock initialized: now 1790722313629562 us; error 33 us; skew 500 ppm
Sep 29 22:51:53 dist-test-slave-98mk krb5kdc[25677](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790722313, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.147.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 22:51:53.638154 26514 init.cc:377] Logged in from keytab as kudu/127.14.147.195@KRBTEST.COM (short username kudu)
I20260929 22:51:53.638903 26514 webserver.cc:460] Webserver started at http://127.14.147.195:44383/ using document root <none> and password file <none>
I20260929 22:51:53.639233 26514 fs_manager.cc:359] Metadata directory not provided
I20260929 22:51:53.639312 26514 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 22:51:53.639521 26514 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 29 22:51:53 dist-test-slave-98mk krb5kdc[25677](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790722313, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.147.195@KRBTEST.COM for HTTP/127.14.147.212@KRBTEST.COM
I20260929 22:51:53.653667 26514 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790722213368738-14927-0/minicluster-data/ts-2/data/instance:
uuid: "5516ad8cb76b40cdba854d3b2b00736c"
format_stamp: "Formatted at 2026-09-29 22:51:53 on dist-test-slave-98mk"
server_key: "aa223b628f65f1b875f180b4109e55e7"
server_key_iv: "12eb4b586f951370ac2dc4d0af502beb"
server_key_version: "kuduclusterkey@0"
I20260929 22:51:53.654134 26514 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790722213368738-14927-0/minicluster-data/ts-2/wal/instance:
uuid: "5516ad8cb76b40cdba854d3b2b00736c"
format_stamp: "Formatted at 2026-09-29 22:51:53 on dist-test-slave-98mk"
server_key: "aa223b628f65f1b875f180b4109e55e7"
server_key_iv: "12eb4b586f951370ac2dc4d0af502beb"
server_key_version: "kuduclusterkey@0"
I20260929 22:51:53.662578 26514 fs_manager.cc:691] Time spent creating directory manager: real 0.008s user 0.004s sys 0.000s
I20260929 22:51:53.683550 26529 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 22:51:53.683812 26514 fs_manager.cc:725] Time spent opening block manager: real 0.007s user 0.000s sys 0.001s
I20260929 22:51:53.683905 26514 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790722213368738-14927-0/minicluster-data/ts-2/data,/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790722213368738-14927-0/minicluster-data/ts-2/wal
uuid: "5516ad8cb76b40cdba854d3b2b00736c"
format_stamp: "Formatted at 2026-09-29 22:51:53 on dist-test-slave-98mk"
server_key: "aa223b628f65f1b875f180b4109e55e7"
server_key_iv: "12eb4b586f951370ac2dc4d0af502beb"
server_key_version: "kuduclusterkey@0"
I20260929 22:51:53.684006 26514 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790722213368738-14927-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790722213368738-14927-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790722213368738-14927-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 22:51:53.706948 26514 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 22:51:53.713812 26514 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 22:51:53.714241 26514 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 22:51:53.714576 26514 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 22:51:53.715024 26514 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 22:51:53.715152 26514 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 22:51:53.715234 26514 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 22:51:53.715291 26514 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 22:51:53.721557 26514 rpc_server.cc:307] RPC server started. Bound to: 127.14.147.195:46435
I20260929 22:51:53.724740 26514 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790722213368738-14927-0/minicluster-data/ts-2/data/info.pb
I20260929 22:51:53.726867 14927 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhWOCLk/build/release/bin/kudu as pid 26514
I20260929 22:51:53.726984 14927 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790722213368738-14927-0/minicluster-data/ts-2/wal/instance
I20260929 22:51:53.727387 14927 external_mini_cluster.cc:1428] Setting key 80081148a54fdb925fdbaa9e3ab47fcd
I20260929 22:51:53.733938 26642 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.147.195:46435 every 8 connection(s)
Sep 29 22:51:53 dist-test-slave-98mk krb5kdc[25677](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790722313, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.147.195@KRBTEST.COM for kudu/127.14.147.254@KRBTEST.COM
I20260929 22:51:53.756702 26373 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:51:53.735183 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.147.195:40363 (local address 127.14.147.254:34001)
0929 22:51:53.735240 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0929 22:51:53.735242 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 22:51:53.735510 (+ 268us) server_negotiation.cc:407] Connection header received
0929 22:51:53.736638 (+ 1128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 22:51:53.736640 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 22:51:53.736653 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 22:51:53.736686 (+ 33us) server_negotiation.cc:226] Negotiated authn=SASL
0929 22:51:53.737587 (+ 901us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:51:53.738353 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:51:53.744112 (+ 5759us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:51:53.744444 (+ 332us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:51:53.744497 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 22:51:53.753352 (+ 8855us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 22:51:53.753355 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 22:51:53.753355 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 22:51:53.753365 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0929 22:51:53.754982 (+ 1617us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 22:51:53.755583 (+ 601us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 22:51:53.755584 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 22:51:53.755585 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 22:51:53.755649 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 22:51:53.755923 (+ 274us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 22:51:53.755924 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 22:51:53.755925 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 22:51:53.756078 (+ 153us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 22:51:53.756127 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0929 22:51:53.756515 (+ 388us) server_negotiation.cc:299] Negotiation successful
0929 22:51:53.756604 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260929 22:51:53.757540 26645 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:51:53.735167 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.147.254:34001 (local address 127.14.147.195:40363)
0929 22:51:53.735346 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0929 22:51:53.735386 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0929 22:51:53.736503 (+ 1117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 22:51:53.736888 (+ 385us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 22:51:53.736893 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 22:51:53.737161 (+ 268us) client_negotiation.cc:189] Negotiated authn=SASL
0929 22:51:53.737474 (+ 313us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 22:51:53.737481 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 22:51:53.740774 (+ 3293us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 22:51:53.740775 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 22:51:53.743972 (+ 3197us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 22:51:53.743977 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 22:51:53.744610 (+ 633us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 22:51:53.744611 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 22:51:53.746251 (+ 1640us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 22:51:53.751390 (+ 5139us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 22:51:53.751404 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0929 22:51:53.753191 (+ 1787us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 22:51:53.755094 (+ 1903us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 22:51:53.755106 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 22:51:53.755110 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 22:51:53.755494 (+ 384us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 22:51:53.755710 (+ 216us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 22:51:53.755711 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 22:51:53.755711 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 22:51:53.755857 (+ 146us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 22:51:53.756163 (+ 306us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 22:51:53.756171 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 22:51:53.756356 (+ 185us) client_negotiation.cc:769] Sending connection context
0929 22:51:53.757104 (+ 748us) client_negotiation.cc:240] Negotiation successful
0929 22:51:53.757291 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":138,"thread_start_us":109,"threads_started":1}
I20260929 22:51:53.758242 26643 heartbeater.cc:347] Connected to a master server at 127.14.147.254:34001
I20260929 22:51:53.758339 26643 heartbeater.cc:464] Registering TS with master...
I20260929 22:51:53.758538 26643 heartbeater.cc:511] Master 127.14.147.254:34001 requested a full tablet report, sending...
I20260929 22:51:53.759017 26158 ts_manager.cc:194] Registered new tserver with Master: 5516ad8cb76b40cdba854d3b2b00736c (127.14.147.195:46435)
I20260929 22:51:53.759565 26158 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.147.195@KRBTEST.COM'} at 127.14.147.195:40363
I20260929 22:51:53.761946 14927 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 22:51:53.771832 26373 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:51:53.763715 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59518 (local address 127.14.147.254:34001)
0929 22:51:53.763760 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0929 22:51:53.763762 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 22:51:53.764070 (+ 308us) server_negotiation.cc:407] Connection header received
0929 22:51:53.764181 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 22:51:53.764182 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 22:51:53.764193 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 22:51:53.764220 (+ 27us) server_negotiation.cc:226] Negotiated authn=SASL
0929 22:51:53.764959 (+ 739us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:51:53.765744 (+ 785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:51:53.766849 (+ 1105us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:51:53.767162 (+ 313us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:51:53.767201 (+ 39us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 22:51:53.768806 (+ 1605us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 22:51:53.768808 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 22:51:53.768808 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 22:51:53.768816 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0929 22:51:53.770476 (+ 1660us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 22:51:53.770934 (+ 458us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 22:51:53.770936 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 22:51:53.770936 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 22:51:53.770994 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 22:51:53.771186 (+ 192us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 22:51:53.771186 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 22:51:53.771187 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 22:51:53.771321 (+ 134us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 22:51:53.771395 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0929 22:51:53.771639 (+ 244us) server_negotiation.cc:299] Negotiation successful
0929 22:51:53.771727 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260929 22:51:53.774322 26158 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59518:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 22:51:53.776657 26158 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 22:51:53.792114 26660 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:51:53.785670 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.147.195:46435 (local address 127.0.0.1:42238)
0929 22:51:53.786715 (+ 1045us) negotiation.cc:107] Waiting for socket to connect
0929 22:51:53.786722 (+ 7us) client_negotiation.cc:174] Beginning negotiation
0929 22:51:53.786750 (+ 28us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 22:51:53.787088 (+ 338us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 22:51:53.787102 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 22:51:53.787105 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 22:51:53.787177 (+ 72us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 22:51:53.787179 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 22:51:53.788855 (+ 1676us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 22:51:53.788856 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 22:51:53.790644 (+ 1788us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 22:51:53.790647 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 22:51:53.791773 (+ 1126us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 22:51:53.791774 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 22:51:53.791862 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 22:51:53.791876 (+ 14us) client_negotiation.cc:769] Sending connection context
0929 22:51:53.791943 (+ 67us) client_negotiation.cc:240] Negotiation successful
0929 22:51:53.792001 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1036,"thread_start_us":78,"threads_started":1}
I20260929 22:51:53.792702 26661 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:51:53.785915 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42238 (local address 127.14.147.195:46435)
0929 22:51:53.786859 (+ 944us) server_negotiation.cc:206] Beginning negotiation
0929 22:51:53.786864 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 22:51:53.786874 (+ 10us) server_negotiation.cc:407] Connection header received
0929 22:51:53.786911 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 22:51:53.786913 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 22:51:53.786929 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 22:51:53.786994 (+ 65us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 22:51:53.787251 (+ 257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:51:53.788769 (+ 1518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:51:53.790748 (+ 1979us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:51:53.791586 (+ 838us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:51:53.791674 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 22:51:53.791736 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0929 22:51:53.792567 (+ 831us) server_negotiation.cc:299] Negotiation successful
0929 22:51:53.792620 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":904,"thread_start_us":68,"threads_started":1}
I20260929 22:51:53.793617 26577 tablet_service.cc:1467] Processing CreateTablet for tablet 086fe01b14ab4090877c45adf9322502 (DEFAULT_TABLE table=test-table [id=be48874920e7480f952202e357819c04]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 22:51:53.793993 26577 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 086fe01b14ab4090877c45adf9322502. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 22:51:53.803720 26658 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:51:53.785243 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.147.194:32833 (local address 127.0.0.1:47010)
0929 22:51:53.786414 (+ 1171us) negotiation.cc:107] Waiting for socket to connect
0929 22:51:53.786460 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0929 22:51:53.786520 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 22:51:53.798286 (+ 11766us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 22:51:53.798286 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 22:51:53.798287 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 22:51:53.798320 (+ 33us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 22:51:53.798321 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 22:51:53.799504 (+ 1183us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 22:51:53.799506 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 22:51:53.802212 (+ 2706us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 22:51:53.802215 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 22:51:53.803397 (+ 1182us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 22:51:53.803398 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 22:51:53.803500 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 22:51:53.803505 (+ 5us) client_negotiation.cc:769] Sending connection context
0929 22:51:53.803556 (+ 51us) client_negotiation.cc:240] Negotiation successful
0929 22:51:53.803614 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1139,"mutex_wait_us":1140,"thread_start_us":77,"threads_started":1}
I20260929 22:51:53.804025 26657 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:51:53.785360 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47010 (local address 127.14.147.194:32833)
0929 22:51:53.798152 (+ 12792us) server_negotiation.cc:206] Beginning negotiation
0929 22:51:53.798156 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 22:51:53.798166 (+ 10us) server_negotiation.cc:407] Connection header received
0929 22:51:53.798199 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 22:51:53.798201 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 22:51:53.798214 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 22:51:53.798284 (+ 70us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 22:51:53.798355 (+ 71us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:51:53.799351 (+ 996us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:51:53.802321 (+ 2970us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:51:53.803191 (+ 870us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:51:53.803291 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 22:51:53.803356 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0929 22:51:53.803902 (+ 546us) server_negotiation.cc:299] Negotiation successful
0929 22:51:53.803949 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12750,"thread_start_us":102,"threads_started":1}
I20260929 22:51:53.804162 26659 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:51:53.785621 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55110 (local address 127.14.147.193:44171)
0929 22:51:53.797821 (+ 12200us) server_negotiation.cc:206] Beginning negotiation
0929 22:51:53.797827 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 22:51:53.797843 (+ 16us) server_negotiation.cc:407] Connection header received
0929 22:51:53.797884 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 22:51:53.797886 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 22:51:53.797905 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 22:51:53.798004 (+ 99us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 22:51:53.798388 (+ 384us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:51:53.799541 (+ 1153us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:51:53.802401 (+ 2860us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:51:53.803192 (+ 791us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:51:53.803268 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 22:51:53.803333 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0929 22:51:53.803768 (+ 435us) server_negotiation.cc:299] Negotiation successful
0929 22:51:53.804013 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12151,"thread_start_us":58,"threads_started":1}
I20260929 22:51:53.803722 26656 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:51:53.785442 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.147.193:44171 (local address 127.0.0.1:55110)
0929 22:51:53.786619 (+ 1177us) negotiation.cc:107] Waiting for socket to connect
0929 22:51:53.786626 (+ 7us) client_negotiation.cc:174] Beginning negotiation
0929 22:51:53.786658 (+ 32us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 22:51:53.798136 (+ 11478us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 22:51:53.798138 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 22:51:53.798140 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 22:51:53.798213 (+ 73us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 22:51:53.798215 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 22:51:53.799704 (+ 1489us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 22:51:53.799705 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 22:51:53.800907 (+ 1202us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 22:51:53.800911 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 22:51:53.803371 (+ 2460us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 22:51:53.803372 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 22:51:53.803519 (+ 147us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 22:51:53.803523 (+ 4us) client_negotiation.cc:769] Sending connection context
0929 22:51:53.803573 (+ 50us) client_negotiation.cc:240] Negotiation successful
0929 22:51:53.803626 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1164,"thread_start_us":52,"threads_started":1}
I20260929 22:51:53.805009 26662 tablet_bootstrap.cc:492] T 086fe01b14ab4090877c45adf9322502 P 5516ad8cb76b40cdba854d3b2b00736c: Bootstrap starting.
I20260929 22:51:53.804934 26443 tablet_service.cc:1467] Processing CreateTablet for tablet 086fe01b14ab4090877c45adf9322502 (DEFAULT_TABLE table=test-table [id=be48874920e7480f952202e357819c04]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 22:51:53.805335 26443 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 086fe01b14ab4090877c45adf9322502. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 22:51:53.806556 26662 tablet_bootstrap.cc:654] T 086fe01b14ab4090877c45adf9322502 P 5516ad8cb76b40cdba854d3b2b00736c: Neither blocks nor log segments found. Creating new log.
I20260929 22:51:53.806852 26662 log.cc:824] T 086fe01b14ab4090877c45adf9322502 P 5516ad8cb76b40cdba854d3b2b00736c: Log is configured to *not* fsync() on all Append() calls
I20260929 22:51:53.807673 26662 tablet_bootstrap.cc:492] T 086fe01b14ab4090877c45adf9322502 P 5516ad8cb76b40cdba854d3b2b00736c: No bootstrap required, opened a new log
I20260929 22:51:53.807737 26662 ts_tablet_manager.cc:1397] T 086fe01b14ab4090877c45adf9322502 P 5516ad8cb76b40cdba854d3b2b00736c: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260929 22:51:53.809645 26662 raft_consensus.cc:348] T 086fe01b14ab4090877c45adf9322502 P 5516ad8cb76b40cdba854d3b2b00736c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "adb47a32e0a3483ab2b65bccba6af58e" member_type: VOTER last_known_addr { host: "127.14.147.194" port: 32833 } } peers { permanent_uuid: "5516ad8cb76b40cdba854d3b2b00736c" member_type: VOTER last_known_addr { host: "127.14.147.195" port: 46435 } } peers { permanent_uuid: "78ec93dc79b041ddb17d95d6241351bd" member_type: VOTER last_known_addr { host: "127.14.147.193" port: 44171 } }
I20260929 22:51:53.809808 26662 raft_consensus.cc:374] T 086fe01b14ab4090877c45adf9322502 P 5516ad8cb76b40cdba854d3b2b00736c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 22:51:53.809834 26662 raft_consensus.cc:729] T 086fe01b14ab4090877c45adf9322502 P 5516ad8cb76b40cdba854d3b2b00736c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5516ad8cb76b40cdba854d3b2b00736c, State: Initialized, Role: FOLLOWER
I20260929 22:51:53.809808 26304 tablet_service.cc:1467] Processing CreateTablet for tablet 086fe01b14ab4090877c45adf9322502 (DEFAULT_TABLE table=test-table [id=be48874920e7480f952202e357819c04]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 22:51:53.809949 26662 consensus_queue.cc:260] T 086fe01b14ab4090877c45adf9322502 P 5516ad8cb76b40cdba854d3b2b00736c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "adb47a32e0a3483ab2b65bccba6af58e" member_type: VOTER last_known_addr { host: "127.14.147.194" port: 32833 } } peers { permanent_uuid: "5516ad8cb76b40cdba854d3b2b00736c" member_type: VOTER last_known_addr { host: "127.14.147.195" port: 46435 } } peers { permanent_uuid: "78ec93dc79b041ddb17d95d6241351bd" member_type: VOTER last_known_addr { host: "127.14.147.193" port: 44171 } }
I20260929 22:51:53.810163 26304 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 086fe01b14ab4090877c45adf9322502. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 22:51:53.810200 26662 ts_tablet_manager.cc:1428] T 086fe01b14ab4090877c45adf9322502 P 5516ad8cb76b40cdba854d3b2b00736c: Time spent starting tablet: real 0.002s user 0.000s sys 0.003s
I20260929 22:51:53.810338 26643 heartbeater.cc:503] Master 127.14.147.254:34001 was elected leader, sending a full tablet report...
I20260929 22:51:53.813561 26666 tablet_bootstrap.cc:492] T 086fe01b14ab4090877c45adf9322502 P 78ec93dc79b041ddb17d95d6241351bd: Bootstrap starting.
I20260929 22:51:53.814270 26666 tablet_bootstrap.cc:654] T 086fe01b14ab4090877c45adf9322502 P 78ec93dc79b041ddb17d95d6241351bd: Neither blocks nor log segments found. Creating new log.
I20260929 22:51:53.814659 26666 log.cc:824] T 086fe01b14ab4090877c45adf9322502 P 78ec93dc79b041ddb17d95d6241351bd: Log is configured to *not* fsync() on all Append() calls
I20260929 22:51:53.815510 26666 tablet_bootstrap.cc:492] T 086fe01b14ab4090877c45adf9322502 P 78ec93dc79b041ddb17d95d6241351bd: No bootstrap required, opened a new log
I20260929 22:51:53.815582 26666 ts_tablet_manager.cc:1397] T 086fe01b14ab4090877c45adf9322502 P 78ec93dc79b041ddb17d95d6241351bd: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260929 22:51:53.818008 26666 raft_consensus.cc:348] T 086fe01b14ab4090877c45adf9322502 P 78ec93dc79b041ddb17d95d6241351bd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "adb47a32e0a3483ab2b65bccba6af58e" member_type: VOTER last_known_addr { host: "127.14.147.194" port: 32833 } } peers { permanent_uuid: "5516ad8cb76b40cdba854d3b2b00736c" member_type: VOTER last_known_addr { host: "127.14.147.195" port: 46435 } } peers { permanent_uuid: "78ec93dc79b041ddb17d95d6241351bd" member_type: VOTER last_known_addr { host: "127.14.147.193" port: 44171 } }
I20260929 22:51:53.818192 26666 raft_consensus.cc:374] T 086fe01b14ab4090877c45adf9322502 P 78ec93dc79b041ddb17d95d6241351bd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 22:51:53.818220 26666 raft_consensus.cc:729] T 086fe01b14ab4090877c45adf9322502 P 78ec93dc79b041ddb17d95d6241351bd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 78ec93dc79b041ddb17d95d6241351bd, State: Initialized, Role: FOLLOWER
I20260929 22:51:53.818876 26666 consensus_queue.cc:260] T 086fe01b14ab4090877c45adf9322502 P 78ec93dc79b041ddb17d95d6241351bd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "adb47a32e0a3483ab2b65bccba6af58e" member_type: VOTER last_known_addr { host: "127.14.147.194" port: 32833 } } peers { permanent_uuid: "5516ad8cb76b40cdba854d3b2b00736c" member_type: VOTER last_known_addr { host: "127.14.147.195" port: 46435 } } peers { permanent_uuid: "78ec93dc79b041ddb17d95d6241351bd" member_type: VOTER last_known_addr { host: "127.14.147.193" port: 44171 } }
I20260929 22:51:53.819253 26666 ts_tablet_manager.cc:1428] T 086fe01b14ab4090877c45adf9322502 P 78ec93dc79b041ddb17d95d6241351bd: Time spent starting tablet: real 0.004s user 0.003s sys 0.000s
I20260929 22:51:53.819458 26370 heartbeater.cc:503] Master 127.14.147.254:34001 was elected leader, sending a full tablet report...
I20260929 22:51:53.819643 26664 tablet_bootstrap.cc:492] T 086fe01b14ab4090877c45adf9322502 P adb47a32e0a3483ab2b65bccba6af58e: Bootstrap starting.
I20260929 22:51:53.820642 26664 tablet_bootstrap.cc:654] T 086fe01b14ab4090877c45adf9322502 P adb47a32e0a3483ab2b65bccba6af58e: Neither blocks nor log segments found. Creating new log.
I20260929 22:51:53.821007 26664 log.cc:824] T 086fe01b14ab4090877c45adf9322502 P adb47a32e0a3483ab2b65bccba6af58e: Log is configured to *not* fsync() on all Append() calls
I20260929 22:51:53.821790 26664 tablet_bootstrap.cc:492] T 086fe01b14ab4090877c45adf9322502 P adb47a32e0a3483ab2b65bccba6af58e: No bootstrap required, opened a new log
I20260929 22:51:53.821854 26664 ts_tablet_manager.cc:1397] T 086fe01b14ab4090877c45adf9322502 P adb47a32e0a3483ab2b65bccba6af58e: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260929 22:51:53.823933 26664 raft_consensus.cc:348] T 086fe01b14ab4090877c45adf9322502 P adb47a32e0a3483ab2b65bccba6af58e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "adb47a32e0a3483ab2b65bccba6af58e" member_type: VOTER last_known_addr { host: "127.14.147.194" port: 32833 } } peers { permanent_uuid: "5516ad8cb76b40cdba854d3b2b00736c" member_type: VOTER last_known_addr { host: "127.14.147.195" port: 46435 } } peers { permanent_uuid: "78ec93dc79b041ddb17d95d6241351bd" member_type: VOTER last_known_addr { host: "127.14.147.193" port: 44171 } }
I20260929 22:51:53.824128 26664 raft_consensus.cc:374] T 086fe01b14ab4090877c45adf9322502 P adb47a32e0a3483ab2b65bccba6af58e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 22:51:53.824160 26664 raft_consensus.cc:729] T 086fe01b14ab4090877c45adf9322502 P adb47a32e0a3483ab2b65bccba6af58e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: adb47a32e0a3483ab2b65bccba6af58e, State: Initialized, Role: FOLLOWER
I20260929 22:51:53.824275 26664 consensus_queue.cc:260] T 086fe01b14ab4090877c45adf9322502 P adb47a32e0a3483ab2b65bccba6af58e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "adb47a32e0a3483ab2b65bccba6af58e" member_type: VOTER last_known_addr { host: "127.14.147.194" port: 32833 } } peers { permanent_uuid: "5516ad8cb76b40cdba854d3b2b00736c" member_type: VOTER last_known_addr { host: "127.14.147.195" port: 46435 } } peers { permanent_uuid: "78ec93dc79b041ddb17d95d6241351bd" member_type: VOTER last_known_addr { host: "127.14.147.193" port: 44171 } }
I20260929 22:51:53.824528 26664 ts_tablet_manager.cc:1428] T 086fe01b14ab4090877c45adf9322502 P adb47a32e0a3483ab2b65bccba6af58e: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260929 22:51:53.824765 26509 heartbeater.cc:503] Master 127.14.147.254:34001 was elected leader, sending a full tablet report...
I20260929 22:51:53.826675 26670 raft_consensus.cc:482] T 086fe01b14ab4090877c45adf9322502 P adb47a32e0a3483ab2b65bccba6af58e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 22:51:53.826819 26670 raft_consensus.cc:504] T 086fe01b14ab4090877c45adf9322502 P adb47a32e0a3483ab2b65bccba6af58e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "adb47a32e0a3483ab2b65bccba6af58e" member_type: VOTER last_known_addr { host: "127.14.147.194" port: 32833 } } peers { permanent_uuid: "5516ad8cb76b40cdba854d3b2b00736c" member_type: VOTER last_known_addr { host: "127.14.147.195" port: 46435 } } peers { permanent_uuid: "78ec93dc79b041ddb17d95d6241351bd" member_type: VOTER last_known_addr { host: "127.14.147.193" port: 44171 } }
I20260929 22:51:53.831230 26512 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:51:53.827329 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.147.195:46435 (local address 127.14.147.194:57877)
0929 22:51:53.827376 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0929 22:51:53.827387 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0929 22:51:53.827446 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 22:51:53.827962 (+ 516us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 22:51:53.827963 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 22:51:53.827965 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 22:51:53.828029 (+ 64us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 22:51:53.828031 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 22:51:53.829010 (+ 979us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 22:51:53.829011 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 22:51:53.830207 (+ 1196us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 22:51:53.830210 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 22:51:53.830952 (+ 742us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 22:51:53.830953 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 22:51:53.831038 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 22:51:53.831043 (+ 5us) client_negotiation.cc:769] Sending connection context
0929 22:51:53.831102 (+ 59us) client_negotiation.cc:240] Negotiation successful
0929 22:51:53.831149 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":26}
I20260929 22:51:53.831970 26661 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:51:53.827525 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.147.194:57877 (local address 127.14.147.195:46435)
0929 22:51:53.827706 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0929 22:51:53.827708 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 22:51:53.827716 (+ 8us) server_negotiation.cc:407] Connection header received
0929 22:51:53.827742 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 22:51:53.827743 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 22:51:53.827751 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 22:51:53.827800 (+ 49us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 22:51:53.828117 (+ 317us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:51:53.828894 (+ 777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:51:53.830309 (+ 1415us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:51:53.830836 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:51:53.830908 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 22:51:53.830918 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0929 22:51:53.831850 (+ 932us) server_negotiation.cc:299] Negotiation successful
0929 22:51:53.831899 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":154}
I20260929 22:51:53.832243 26597 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "086fe01b14ab4090877c45adf9322502" candidate_uuid: "adb47a32e0a3483ab2b65bccba6af58e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5516ad8cb76b40cdba854d3b2b00736c" is_pre_election: true
I20260929 22:51:53.832407 26597 raft_consensus.cc:2454] T 086fe01b14ab4090877c45adf9322502 P 5516ad8cb76b40cdba854d3b2b00736c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate adb47a32e0a3483ab2b65bccba6af58e in term 0.
I20260929 22:51:53.832664 26394 leader_election.cc:302] T 086fe01b14ab4090877c45adf9322502 P adb47a32e0a3483ab2b65bccba6af58e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5516ad8cb76b40cdba854d3b2b00736c, adb47a32e0a3483ab2b65bccba6af58e; no voters:
I20260929 22:51:53.832816 26671 raft_consensus.cc:2781] T 086fe01b14ab4090877c45adf9322502 P adb47a32e0a3483ab2b65bccba6af58e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 22:51:53.832856 26671 raft_consensus.cc:482] T 086fe01b14ab4090877c45adf9322502 P adb47a32e0a3483ab2b65bccba6af58e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 22:51:53.832875 26671 raft_consensus.cc:3037] T 086fe01b14ab4090877c45adf9322502 P adb47a32e0a3483ab2b65bccba6af58e [term 0 FOLLOWER]: Advancing to term 1
I20260929 22:51:53.833788 26671 raft_consensus.cc:504] T 086fe01b14ab4090877c45adf9322502 P adb47a32e0a3483ab2b65bccba6af58e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "adb47a32e0a3483ab2b65bccba6af58e" member_type: VOTER last_known_addr { host: "127.14.147.194" port: 32833 } } peers { permanent_uuid: "5516ad8cb76b40cdba854d3b2b00736c" member_type: VOTER last_known_addr { host: "127.14.147.195" port: 46435 } } peers { permanent_uuid: "78ec93dc79b041ddb17d95d6241351bd" member_type: VOTER last_known_addr { host: "127.14.147.193" port: 44171 } }
I20260929 22:51:53.834079 26597 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "086fe01b14ab4090877c45adf9322502" candidate_uuid: "adb47a32e0a3483ab2b65bccba6af58e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5516ad8cb76b40cdba854d3b2b00736c"
I20260929 22:51:53.834151 26597 raft_consensus.cc:3037] T 086fe01b14ab4090877c45adf9322502 P 5516ad8cb76b40cdba854d3b2b00736c [term 0 FOLLOWER]: Advancing to term 1
I20260929 22:51:53.835009 26597 raft_consensus.cc:2454] T 086fe01b14ab4090877c45adf9322502 P 5516ad8cb76b40cdba854d3b2b00736c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate adb47a32e0a3483ab2b65bccba6af58e in term 1.
I20260929 22:51:53.835179 26394 leader_election.cc:302] T 086fe01b14ab4090877c45adf9322502 P adb47a32e0a3483ab2b65bccba6af58e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5516ad8cb76b40cdba854d3b2b00736c, adb47a32e0a3483ab2b65bccba6af58e; no voters:
I20260929 22:51:53.835280 26512 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:51:53.827547 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.147.193:44171 (local address 127.14.147.194:34519)
0929 22:51:53.831309 (+ 3762us) negotiation.cc:107] Waiting for socket to connect
0929 22:51:53.831320 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0929 22:51:53.831360 (+ 40us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 22:51:53.832219 (+ 859us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 22:51:53.832220 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 22:51:53.832222 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 22:51:53.832260 (+ 38us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 22:51:53.832262 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 22:51:53.833247 (+ 985us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 22:51:53.833248 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 22:51:53.834196 (+ 948us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 22:51:53.834199 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 22:51:53.835059 (+ 860us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 22:51:53.835060 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 22:51:53.835133 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 22:51:53.835137 (+ 4us) client_negotiation.cc:769] Sending connection context
0929 22:51:53.835179 (+ 42us) client_negotiation.cc:240] Negotiation successful
0929 22:51:53.835218 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3750,"thread_start_us":68,"threads_started":1}
I20260929 22:51:53.835816 26659 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:51:53.827894 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.147.194:34519 (local address 127.14.147.193:44171)
0929 22:51:53.827932 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0929 22:51:53.827933 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0929 22:51:53.831458 (+ 3525us) server_negotiation.cc:407] Connection header received
0929 22:51:53.831489 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 22:51:53.831490 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 22:51:53.831501 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 22:51:53.831542 (+ 41us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 22:51:53.832347 (+ 805us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:51:53.832984 (+ 637us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:51:53.834372 (+ 1388us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:51:53.834945 (+ 573us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:51:53.835016 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 22:51:53.835027 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0929 22:51:53.835695 (+ 668us) server_negotiation.cc:299] Negotiation successful
0929 22:51:53.835748 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260929 22:51:53.836066 26320 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "086fe01b14ab4090877c45adf9322502" candidate_uuid: "adb47a32e0a3483ab2b65bccba6af58e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "78ec93dc79b041ddb17d95d6241351bd" is_pre_election: true
I20260929 22:51:53.836208 26320 raft_consensus.cc:2454] T 086fe01b14ab4090877c45adf9322502 P 78ec93dc79b041ddb17d95d6241351bd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate adb47a32e0a3483ab2b65bccba6af58e in term 0.
I20260929 22:51:53.836992 26670 leader_election.cc:288] T 086fe01b14ab4090877c45adf9322502 P adb47a32e0a3483ab2b65bccba6af58e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5516ad8cb76b40cdba854d3b2b00736c (127.14.147.195:46435), 78ec93dc79b041ddb17d95d6241351bd (127.14.147.193:44171)
I20260929 22:51:53.837019 26320 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "086fe01b14ab4090877c45adf9322502" candidate_uuid: "adb47a32e0a3483ab2b65bccba6af58e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "78ec93dc79b041ddb17d95d6241351bd"
I20260929 22:51:53.837101 26320 raft_consensus.cc:3037] T 086fe01b14ab4090877c45adf9322502 P 78ec93dc79b041ddb17d95d6241351bd [term 0 FOLLOWER]: Advancing to term 1
I20260929 22:51:53.837239 26670 raft_consensus.cc:2781] T 086fe01b14ab4090877c45adf9322502 P adb47a32e0a3483ab2b65bccba6af58e [term 1 FOLLOWER]: Leader election won for term 1
I20260929 22:51:53.837316 26670 raft_consensus.cc:686] T 086fe01b14ab4090877c45adf9322502 P adb47a32e0a3483ab2b65bccba6af58e [term 1 LEADER]: Becoming Leader. State: Replica: adb47a32e0a3483ab2b65bccba6af58e, State: Running, Role: LEADER
I20260929 22:51:53.837430 26670 consensus_queue.cc:237] T 086fe01b14ab4090877c45adf9322502 P adb47a32e0a3483ab2b65bccba6af58e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "adb47a32e0a3483ab2b65bccba6af58e" member_type: VOTER last_known_addr { host: "127.14.147.194" port: 32833 } } peers { permanent_uuid: "5516ad8cb76b40cdba854d3b2b00736c" member_type: VOTER last_known_addr { host: "127.14.147.195" port: 46435 } } peers { permanent_uuid: "78ec93dc79b041ddb17d95d6241351bd" member_type: VOTER last_known_addr { host: "127.14.147.193" port: 44171 } }
I20260929 22:51:53.838018 26320 raft_consensus.cc:2454] T 086fe01b14ab4090877c45adf9322502 P 78ec93dc79b041ddb17d95d6241351bd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate adb47a32e0a3483ab2b65bccba6af58e in term 1.
I20260929 22:51:53.838423 26158 catalog_manager.cc:5495] T 086fe01b14ab4090877c45adf9322502 P adb47a32e0a3483ab2b65bccba6af58e reported cstate change: term changed from 0 to 1, leader changed from <none> to adb47a32e0a3483ab2b65bccba6af58e (127.14.147.194). New cstate: current_term: 1 leader_uuid: "adb47a32e0a3483ab2b65bccba6af58e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "adb47a32e0a3483ab2b65bccba6af58e" member_type: VOTER last_known_addr { host: "127.14.147.194" port: 32833 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5516ad8cb76b40cdba854d3b2b00736c" member_type: VOTER last_known_addr { host: "127.14.147.195" port: 46435 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "78ec93dc79b041ddb17d95d6241351bd" member_type: VOTER last_known_addr { host: "127.14.147.193" port: 44171 } health_report { overall_health: UNKNOWN } } }
I20260929 22:51:53.838797 26671 leader_election.cc:288] T 086fe01b14ab4090877c45adf9322502 P adb47a32e0a3483ab2b65bccba6af58e [CANDIDATE]: Term 1 election: Requested vote from peers 5516ad8cb76b40cdba854d3b2b00736c (127.14.147.195:46435), 78ec93dc79b041ddb17d95d6241351bd (127.14.147.193:44171)
Sep 29 22:51:53 dist-test-slave-98mk krb5kdc[25677](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790722313, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.147.254@KRBTEST.COM for HTTP/127.14.147.212@KRBTEST.COM
I20260929 22:51:53.859508 26657 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:51:53.855891 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47022 (local address 127.14.147.194:32833)
0929 22:51:53.855937 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0929 22:51:53.855939 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 22:51:53.855949 (+ 10us) server_negotiation.cc:407] Connection header received
0929 22:51:53.856024 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 22:51:53.856025 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 22:51:53.856049 (+ 24us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 22:51:53.856139 (+ 90us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 22:51:53.856336 (+ 197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:51:53.857098 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:51:53.858265 (+ 1167us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:51:53.858610 (+ 345us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:51:53.858666 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 22:51:53.858889 (+ 223us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 22:51:53.859034 (+ 145us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 22:51:53.859110 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0929 22:51:53.859346 (+ 236us) server_negotiation.cc:299] Negotiation successful
0929 22:51:53.859370 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260929 22:51:53.872817 26597 raft_consensus.cc:1261] T 086fe01b14ab4090877c45adf9322502 P 5516ad8cb76b40cdba854d3b2b00736c [term 1 FOLLOWER]: Refusing update from remote peer adb47a32e0a3483ab2b65bccba6af58e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 22:51:53.873231 26320 raft_consensus.cc:1261] T 086fe01b14ab4090877c45adf9322502 P 78ec93dc79b041ddb17d95d6241351bd [term 1 FOLLOWER]: Refusing update from remote peer adb47a32e0a3483ab2b65bccba6af58e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 22:51:53.873615 26671 consensus_queue.cc:1035] T 086fe01b14ab4090877c45adf9322502 P adb47a32e0a3483ab2b65bccba6af58e [LEADER]: Connected to new peer: Peer: permanent_uuid: "5516ad8cb76b40cdba854d3b2b00736c" member_type: VOTER last_known_addr { host: "127.14.147.195" port: 46435 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 22:51:53.874013 26671 consensus_queue.cc:1035] T 086fe01b14ab4090877c45adf9322502 P adb47a32e0a3483ab2b65bccba6af58e [LEADER]: Connected to new peer: Peer: permanent_uuid: "78ec93dc79b041ddb17d95d6241351bd" member_type: VOTER last_known_addr { host: "127.14.147.193" port: 44171 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 22:51:53.886564 26676 mvcc.cc:205] Tried to move back new op lower bound from 7334798597619167232 to 7334798597479043072. Current Snapshot: MvccSnapshot[applied={T|T < 7334798597619167232}]
I20260929 22:51:53.889212 26677 mvcc.cc:205] Tried to move back new op lower bound from 7334798597619167232 to 7334798597479043072. Current Snapshot: MvccSnapshot[applied={T|T < 7334798597619167232}]
I20260929 22:51:53.890497 26679 mvcc.cc:205] Tried to move back new op lower bound from 7334798597619167232 to 7334798597479043072. Current Snapshot: MvccSnapshot[applied={T|T < 7334798597619167232}]
I20260929 22:51:53.894565 26158 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59518:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 22:51:53.894699 26158 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59518:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 22:51:53.895710 26158 catalog_manager.cc:5782] T 00000000000000000000000000000000 P fd0f765e325e4140bf9771ccb19f3e4a: Sending DeleteTablet for 3 replicas of tablet 086fe01b14ab4090877c45adf9322502
I20260929 22:51:53.896092 26304 tablet_service.cc:1514] Processing DeleteTablet for tablet 086fe01b14ab4090877c45adf9322502 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 22:51:53 UTC) from {username='kudu', principal='kudu/127.14.147.254@KRBTEST.COM'} at 127.0.0.1:55110
I20260929 22:51:53.896503 26577 tablet_service.cc:1514] Processing DeleteTablet for tablet 086fe01b14ab4090877c45adf9322502 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 22:51:53 UTC) from {username='kudu', principal='kudu/127.14.147.254@KRBTEST.COM'} at 127.0.0.1:42238
I20260929 22:51:53.896898 26443 tablet_service.cc:1514] Processing DeleteTablet for tablet 086fe01b14ab4090877c45adf9322502 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 22:51:53 UTC) from {username='kudu', principal='kudu/127.14.147.254@KRBTEST.COM'} at 127.0.0.1:47010
I20260929 22:51:53.911825 14927 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhWOCLk/build/release/bin/kudu with pid 26239
I20260929 22:51:53.923913 26686 tablet_replica.cc:330] T 086fe01b14ab4090877c45adf9322502 P 5516ad8cb76b40cdba854d3b2b00736c: stopping tablet replica
I20260929 22:51:53.924108 26686 raft_consensus.cc:2229] T 086fe01b14ab4090877c45adf9322502 P 5516ad8cb76b40cdba854d3b2b00736c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 22:51:53.924261 26686 raft_consensus.cc:2258] T 086fe01b14ab4090877c45adf9322502 P 5516ad8cb76b40cdba854d3b2b00736c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 22:51:53.924453 26687 tablet_replica.cc:330] T 086fe01b14ab4090877c45adf9322502 P adb47a32e0a3483ab2b65bccba6af58e: stopping tablet replica
I20260929 22:51:53.924579 26687 raft_consensus.cc:2229] T 086fe01b14ab4090877c45adf9322502 P adb47a32e0a3483ab2b65bccba6af58e [term 1 LEADER]: Raft consensus shutting down.
I20260929 22:51:53.924736 26687 raft_consensus.cc:2258] T 086fe01b14ab4090877c45adf9322502 P adb47a32e0a3483ab2b65bccba6af58e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 22:51:53.925241 26687 ts_tablet_manager.cc:1905] T 086fe01b14ab4090877c45adf9322502 P adb47a32e0a3483ab2b65bccba6af58e: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260929 22:51:53.931787 26144 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.14.147.193:44171 (error 108)
I20260929 22:51:53.932122 26687 ts_tablet_manager.cc:1918] T 086fe01b14ab4090877c45adf9322502 P adb47a32e0a3483ab2b65bccba6af58e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 22:51:53.932322 26687 log.cc:1196] T 086fe01b14ab4090877c45adf9322502 P adb47a32e0a3483ab2b65bccba6af58e: Deleting WAL directory at /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790722213368738-14927-0/minicluster-data/ts-1/wal/wals/086fe01b14ab4090877c45adf9322502
I20260929 22:51:53.932646 26687 ts_tablet_manager.cc:1939] T 086fe01b14ab4090877c45adf9322502 P adb47a32e0a3483ab2b65bccba6af58e: Deleting consensus metadata
I20260929 22:51:53.933179 26144 catalog_manager.cc:4841] TS adb47a32e0a3483ab2b65bccba6af58e (127.14.147.194:32833): tablet 086fe01b14ab4090877c45adf9322502 (table test-table [id=be48874920e7480f952202e357819c04]) successfully deleted
I20260929 22:51:53.933395 14927 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhWOCLk/build/release/bin/kudu with pid 26376
I20260929 22:51:53.934859 26686 ts_tablet_manager.cc:1905] T 086fe01b14ab4090877c45adf9322502 P 5516ad8cb76b40cdba854d3b2b00736c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 22:51:53.936939 26686 ts_tablet_manager.cc:1918] T 086fe01b14ab4090877c45adf9322502 P 5516ad8cb76b40cdba854d3b2b00736c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 22:51:53.937039 26686 log.cc:1196] T 086fe01b14ab4090877c45adf9322502 P 5516ad8cb76b40cdba854d3b2b00736c: Deleting WAL directory at /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790722213368738-14927-0/minicluster-data/ts-2/wal/wals/086fe01b14ab4090877c45adf9322502
I20260929 22:51:53.937602 26686 ts_tablet_manager.cc:1939] T 086fe01b14ab4090877c45adf9322502 P 5516ad8cb76b40cdba854d3b2b00736c: Deleting consensus metadata
I20260929 22:51:53.938112 26141 catalog_manager.cc:4841] TS 5516ad8cb76b40cdba854d3b2b00736c (127.14.147.195:46435): tablet 086fe01b14ab4090877c45adf9322502 (table test-table [id=be48874920e7480f952202e357819c04]) successfully deleted
I20260929 22:51:53.939560 14927 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhWOCLk/build/release/bin/kudu with pid 26514
I20260929 22:51:53.941582 26656 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:51:53.941438 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.147.193:44171 (local address 127.0.0.1:55122)
0929 22:51:53.941487 (+ 49us) negotiation.cc:107] Waiting for socket to connect
0929 22:51:53.941541 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.147.193:44171: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":24}
W20260929 22:51:53.941727 26144 catalog_manager.cc:4584] TS 78ec93dc79b041ddb17d95d6241351bd (127.14.147.193:44171): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 086fe01b14ab4090877c45adf9322502: Network error: Client connection negotiation failed: client connection to 127.14.147.193:44171: connect: Connection refused (error 111)
I20260929 22:51:53.945508 14927 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhWOCLk/build/release/bin/kudu with pid 26125
I20260929 22:51:53.950116 14927 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260929 22:51:54.284832 14927 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260929 22:51:54.284945 14927 mini_ranger.cc:66] Stopping Ranger...
I20260929 22:51:54.620154 14927 mini_ranger.cc:68] Stopped Ranger
2026-09-29 22:51:54.620 UTC [25707] LOG: received smart shutdown request
2026-09-29 22:51:54.623 UTC [25707] LOG: background worker "logical replication launcher" (PID 25716) exited with exit code 1
2026-09-29 22:51:54.624 UTC [25711] LOG: shutting down
2026-09-29 22:51:54.771 UTC [25707] LOG: database system is shut down
2026-09-29T22:51:54Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegration (47313 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers
Loading random data
Initializing database '/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790722213368738-14927-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 22:51:54 dist-test-slave-98mk krb5kdc[26697](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 22:51:54 dist-test-slave-98mk krb5kdc[26697](info): set up 2 sockets
Sep 29 22:51:54 dist-test-slave-98mk krb5kdc[26697](info): commencing operation
krb5kdc: starting...
W20260929 22:51:55.803943 14927 mini_kdc.cc:121] Time spent starting KDC: real 0.931s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790722213368738-14927-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790722213368738-14927-0/krb5kdc/test-user.keytab.
Sep 29 22:51:55 dist-test-slave-98mk krb5kdc[26697](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790722315, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T22:51:55Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T22:51:55Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.14.147.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.14.147.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.14.147.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790722213368738-14927-0/krb5kdc/rangeradmin_127.14.147.212@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.14.147.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790722213368738-14927-0/krb5kdc/rangeradmin_127.14.147.212@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.14.147.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.14.147.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.14.147.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790722213368738-14927-0/krb5kdc/rangerlookup_127.14.147.212@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.14.147.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790722213368738-14927-0/krb5kdc/rangerlookup_127.14.147.212@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.14.147.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.14.147.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.14.147.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790722213368738-14927-0/krb5kdc/HTTP_127.14.147.212@KRBTEST.COM.keytab.
Entry for principal HTTP/127.14.147.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790722213368738-14927-0/krb5kdc/HTTP_127.14.147.212@KRBTEST.COM.keytab.
I20260929 22:51:55.958158 14927 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790722213368738-14927-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskhWOCLk/build/release/bin/postgres/pg_ctl -D /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790722213368738-14927-0/minicluster-data/postgres -l logfile start
2026-09-29 22:51:58.215 UTC [26727] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-09-29 22:51:58.216 UTC [26727] LOG: listening on IPv4 address "127.14.147.212", port 51653
2026-09-29 22:51:58.219 UTC [26727] LOG: listening on Unix socket "/tmp/.s.PGSQL.51653"
2026-09-29 22:51:58.238 UTC [26730] LOG: database system was shut down at 2026-09-29 22:51:57 UTC
2026-09-29 22:51:58.242 UTC [26727] LOG: database system is ready to accept connections
I20260929 22:51:59.115782 14927 mini_postgres.cc:96] Postgres bound to 51653
2026-09-29 22:51:59.121 UTC [26738] FATAL: database "slave" does not exist
127.14.147.212:51653 - accepting connections
I20260929 22:51:59.121939 14927 mini_ranger.cc:161] Starting Ranger...
I20260929 22:51:59.136144 14927 mini_ranger.cc:84] Created miniranger Postgres user
I20260929 22:51:59.544540 14927 mini_ranger.cc:87] Created ranger Postgres database
I20260929 22:51:59.544636 14927 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790722213368738-14927-0/minicluster-data/ranger-admin
2026-09-29 22:51:59,817 [I] DB FLAVOR :POSTGRES
2026-09-29 22:51:59,818 [I] --------- Verifying Ranger DB connection ---------
2026-09-29 22:51:59,818 [I] Checking connection..
2026-09-29 22:51:59,818 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskhWOCLk/build/release/bin/postgresql.jar:/tmp/dist-test-taskhWOCLk/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.14.147.212:51653/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-09-29 22:52:00,069 [I] Checking connection passed.
2026-09-29 22:52:00,070 [I] --------- Verifying version history table ---------
2026-09-29 22:52:00,070 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskhWOCLk/build/release/bin/postgresql.jar:/tmp/dist-test-taskhWOCLk/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.14.147.212:51653/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-29 22:52:00,329 [I] Table x_db_version_h does not exist in database ranger
2026-09-29 22:52:00,330 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskhWOCLk/build/release/bin/postgresql.jar:/tmp/dist-test-taskhWOCLk/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.14.147.212:51653/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-29 22:52:00,578 [I] Table x_db_version_h does not exist in database ranger
2026-09-29 22:52:00,578 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-09-29 22:52:00,578 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskhWOCLk/build/release/bin/postgresql.jar:/tmp/dist-test-taskhWOCLk/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.14.147.212:51653/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskhWOCLk/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-09-29 22:52:00.814 UTC [26802] WARNING: there is no transaction in progress
2026-09-29 22:52:00,820 [I] create_dbversion_catalog.sql file imported successfully
2026-09-29 22:52:00,820 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskhWOCLk/build/release/bin/postgresql.jar:/tmp/dist-test-taskhWOCLk/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.14.147.212:51653/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-29 22:52:01,066 [I] Table x_db_version_h already exists in database 'ranger'
2026-09-29 22:52:01,067 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-29 22:52:01,067 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskhWOCLk/build/release/bin/postgresql.jar:/tmp/dist-test-taskhWOCLk/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.14.147.212:51653/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-09-29 22:52:01,328 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskhWOCLk/build/release/bin/postgresql.jar:/tmp/dist-test-taskhWOCLk/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.14.147.212:51653/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-09-29 22:52:01,566 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskhWOCLk/build/release/bin/postgresql.jar:/tmp/dist-test-taskhWOCLk/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.14.147.212:51653/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-98mk.c.gcp-upstream.internal','N') ;"
2026-09-29 22:52:01,826 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskhWOCLk/build/release/bin/postgresql.jar:/tmp/dist-test-taskhWOCLk/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.14.147.212:51653/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-29 22:52:02,092 [I] Table x_portal_user does not exist in database ranger
2026-09-29 22:52:02,093 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskhWOCLk/build/release/bin/postgresql.jar:/tmp/dist-test-taskhWOCLk/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.14.147.212:51653/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-29 22:52:02,361 [I] Table x_policy_ref_group does not exist in database ranger
2026-09-29 22:52:02,361 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskhWOCLk/build/release/bin/postgresql.jar:/tmp/dist-test-taskhWOCLk/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.14.147.212:51653/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-29 22:52:02,603 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskhWOCLk/build/release/bin/postgresql.jar:/tmp/dist-test-taskhWOCLk/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.14.147.212:51653/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-29 22:52:02,864 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-09-29 22:52:02,864 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskhWOCLk/build/release/bin/postgresql.jar:/tmp/dist-test-taskhWOCLk/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.14.147.212:51653/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskhWOCLk/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-09-29 22:52:03.858 UTC [26910] WARNING: there is no transaction in progress
2026-09-29 22:52:03.874 UTC [26910] WARNING: there is no transaction in progress
2026-09-29 22:52:03.890 UTC [26910] WARNING: there is no transaction in progress
2026-09-29 22:52:03.906 UTC [26910] WARNING: there is no transaction in progress
2026-09-29 22:52:03.923 UTC [26910] WARNING: there is no transaction in progress
2026-09-29 22:52:03.940 UTC [26910] WARNING: there is no transaction in progress
2026-09-29 22:52:03.997 UTC [26910] WARNING: there is no transaction in progress
2026-09-29 22:52:04.003 UTC [26910] WARNING: there is no transaction in progress
2026-09-29 22:52:04.016 UTC [26910] WARNING: there is no transaction in progress
2026-09-29 22:52:04.027 UTC [26910] WARNING: there is no transaction in progress
2026-09-29 22:52:04.037 UTC [26910] WARNING: there is no transaction in progress
2026-09-29 22:52:04.051 UTC [26910] WARNING: there is no transaction in progress
2026-09-29 22:52:04.061 UTC [26910] WARNING: there is no transaction in progress
2026-09-29 22:52:04.068 UTC [26910] WARNING: there is no transaction in progress
2026-09-29 22:52:04.074 UTC [26910] WARNING: there is no transaction in progress
2026-09-29 22:52:04.082 UTC [26910] WARNING: there is no transaction in progress
2026-09-29 22:52:04,708 [I] ranger_core_db_postgres.sql file imported successfully
2026-09-29 22:52:04,708 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskhWOCLk/build/release/bin/postgresql.jar:/tmp/dist-test-taskhWOCLk/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.14.147.212:51653/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-09-29 22:52:04,969 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-09-29 22:52:04,969 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskhWOCLk/build/release/bin/postgresql.jar:/tmp/dist-test-taskhWOCLk/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.14.147.212:51653/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-29 22:52:05,224 [I] Table x_portal_user already exists in database 'ranger'
2026-09-29 22:52:05,224 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskhWOCLk/build/release/bin/postgresql.jar:/tmp/dist-test-taskhWOCLk/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.14.147.212:51653/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-29 22:52:05,493 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-09-29 22:52:05,493 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskhWOCLk/build/release/bin/postgresql.jar:/tmp/dist-test-taskhWOCLk/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.14.147.212:51653/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-29 22:52:05,752 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskhWOCLk/build/release/bin/postgresql.jar:/tmp/dist-test-taskhWOCLk/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.14.147.212:51653/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-29 22:52:05,989 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskhWOCLk/build/release/bin/postgresql.jar:/tmp/dist-test-taskhWOCLk/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.14.147.212:51653/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-98mk.c.gcp-upstream.internal';"
2026-09-29 22:52:06,242 [I] CORE_DB_SCHEMA import status has been updated
2026-09-29 22:52:06,243 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskhWOCLk/build/release/bin/postgresql.jar:/tmp/dist-test-taskhWOCLk/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.14.147.212:51653/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-09-29 22:52:06,506 [I] DB_PATCHES have already been applied
I20260929 22:52:06.511972 14927 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790722213368738-14927-0/minicluster-data/ranger-admin:/tmp/dist-test-taskhWOCLk/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskhWOCLk/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskhWOCLk/build/release/bin/postgresql.jar:/tmp/dist-test-taskhWOCLk/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260929 22:52:06.512048 14927 mini_ranger.cc:193] Using host: 127.14.147.212
I20260929 22:52:06.515126 14927 mini_ranger.cc:227] Ranger admin URL: http://127.14.147.212:54149
Sep 29, 2026 10:52:06 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 29, 2026 10:52:06 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 29, 2026 10:52:06 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790722213368738-14927-0/minicluster-data/ranger-admin/ews/webapp
Sep 29, 2026 10:52:06 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790722213368738-14927-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Sep 29, 2026 10:52:06 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790722213368738-14927-0/minicluster-data/ranger-admin/ews/webapp] .....
Sep 29, 2026 10:52:07 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790722213368738-14927-0/minicluster-data/ranger-admin/ews/webapp].
Sep 29, 2026 10:52:07 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260929 22:52:07.526492 14927 mini_ranger.cc:160] Time spent starting Ranger: real 8.405s user 0.000s sys 0.008s
Sep 29, 2026 10:52:28 PM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Sep 29, 2026 10:52:28 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.UserREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
Sep 29, 2026 10:52:28 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Sep 29, 2026 10:52:28 PM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Sep 29, 2026 10:52:28 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Sep 29, 2026 10:52:28 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Sep 29, 2026 10:52:28 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Sep 29, 2026 10:52:28 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Sep 29, 2026 10:52:28 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Sep 29, 2026 10:52:28 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Sep 29, 2026 10:52:28 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Sep 29, 2026 10:52:28 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Sep 29, 2026 10:52:28 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Sep 29, 2026 10:52:28 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Sep 29, 2026 10:52:28 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Sep 29, 2026 10:52:28 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Sep 29, 2026 10:52:28 PM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Sep 29, 2026 10:52:29 PM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260929 22:52:30.042879 14927 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.14.147.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.14.147.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.14.147.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790722213368738-14927-0/krb5kdc/rangerkms_127.14.147.212@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.14.147.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790722213368738-14927-0/krb5kdc/rangerkms_127.14.147.212@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.14.147.212@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.14.147.212@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.14.147.212@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790722213368738-14927-0/krb5kdc/HTTP_127.14.147.212@KRBTEST.COM.keytab.
Entry for principal HTTP/127.14.147.212@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790722213368738-14927-0/krb5kdc/HTTP_127.14.147.212@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Sep 29 22:52:30 dist-test-slave-98mk krb5kdc[26697](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790722350, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260929 22:52:30.138702 14927 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260929 22:52:30.149708 14927 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260929 22:52:30.883570 14927 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260929 22:52:30.883637 14927 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790722213368738-14927-0/minicluster-data/ranger-kms
I20260929 22:52:30.883648 14927 mini_ranger_kms.cc:227] Using postgres at 127.14.147.212:51653
2026-09-29 22:52:31,037 [I] DB FLAVOR :POSTGRES
2026-09-29 22:52:31,037 [I] --------- Verifying Ranger DB connection ---------
2026-09-29 22:52:31,037 [I] Checking connection
2026-09-29 22:52:31,037 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskhWOCLk/build/release/bin/postgresql.jar:/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790722213368738-14927-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.14.147.212:51653/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-09-29 22:52:31,285 [I] connection success
2026-09-29 22:52:31,285 [I] --------- Verifying Ranger DB tables ---------
2026-09-29 22:52:31,285 [I] Verifying table ranger_masterkey in database rangerkms
2026-09-29 22:52:31,286 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskhWOCLk/build/release/bin/postgresql.jar:/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790722213368738-14927-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.14.147.212:51653/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-09-29 22:52:31,532 [I] Table ranger_masterkey does not exist in database rangerkms
2026-09-29 22:52:31,532 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-29 22:52:31,532 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-09-29 22:52:31,533 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskhWOCLk/build/release/bin/postgresql.jar:/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790722213368738-14927-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.14.147.212:51653/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790722213368738-14927-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-09-29 22:52:31,787 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260929 22:52:32.288888 14927 mini_ranger_kms.cc:310] Created kms service
I20260929 22:52:32.455251 14927 mini_ranger_kms.cc:326] Created kudu user
I20260929 22:52:32.498492 14927 mini_ranger_kms.cc:343] Created rangerkms user
Sep 29 22:52:32 dist-test-slave-98mk krb5kdc[26697](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790722350, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.14.147.212@KRBTEST.COM
I20260929 22:52:32.961692 14927 mini_ranger_kms.cc:384] Added ranger policy
I20260929 22:52:32.961804 14927 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790722213368738-14927-0/minicluster-data/ranger-kms:/tmp/dist-test-taskhWOCLk/build/release/bin/postgresql.jar:/tmp/dist-test-taskhWOCLk/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskhWOCLk/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260929 22:52:32.961822 14927 mini_ranger_kms.cc:240] Using host: 127.14.147.212
I20260929 22:52:32.964877 14927 mini_ranger_kms.cc:276] Ranger KMS PID: 27090
I20260929 22:52:32.964958 14927 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.14.147.212:46007
Sep 29, 2026 10:52:34 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 29, 2026 10:52:34 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 29, 2026 10:52:34 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790722213368738-14927-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Sep 29, 2026 10:52:34 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790722213368738-14927-0/minicluster-data/ranger-kms/ews/webapp] .....
Sep 29, 2026 10:52:34 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790722213368738-14927-0/minicluster-data/ranger-kms/ews/webapp].
Sep 29, 2026 10:52:34 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260929 22:52:35.046780 14927 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 4.908s user 0.006s sys 0.130s
I20260929 22:52:35.046865 14927 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260929 22:52:35.046909 14927 mini_ranger_kms.cc:401] 127.14.147.212:46007/kms/v1/keys
Sep 29 22:52:35 dist-test-slave-98mk krb5kdc[26697](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790722355, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.14.147.212@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Sep 29 22:52:35 dist-test-slave-98mk krb5kdc[26697](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790722355, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.14.147.212@KRBTEST.COM for HTTP/127.14.147.212@KRBTEST.COM
Sep 29 22:52:36 dist-test-slave-98mk krb5kdc[26697](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790722355, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.14.147.212@KRBTEST.COM for HTTP/127.14.147.212@KRBTEST.COM
Sep 29 22:52:38 dist-test-slave-98mk krb5kdc[26697](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790722358, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.14.147.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.147.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.147.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790722213368738-14927-0/krb5kdc/kudu_127.14.147.254.keytab.
Entry for principal kudu/127.14.147.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790722213368738-14927-0/krb5kdc/kudu_127.14.147.254.keytab.
I20260929 22:52:38.744927 14927 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhWOCLk/build/release/bin/kudu
/tmp/dist-test-taskhWOCLk/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790722213368738-14927-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790722213368738-14927-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790722213368738-14927-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790722213368738-14927-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.147.254:37205
--webserver_interface=127.14.147.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790722213368738-14927-0/krb5kdc/kudu_127.14.147.254.keytab
--principal=kudu/127.14.147.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.147.212:33235
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.147.254:37205
--ranger_config_path=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790722213368738-14927-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.14.147.212:46007/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790722213368738-14927-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790722213368738-14927-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790722213368738-14927-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 22:52:38.839926 27141 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 22:52:38.840142 27141 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 22:52:38.840164 27141 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 22:52:38.841437 27141 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 22:52:38.841488 27141 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 22:52:38.841501 27141 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 22:52:38.841512 27141 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 22:52:38.841523 27141 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 22:52:38.843199 27141 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.147.212:33235
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790722213368738-14927-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790722213368738-14927-0/minicluster-data/master-0/wal
--ranger_kms_url=127.14.147.212:46007/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.14.147.254:37205
--ranger_config_path=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790722213368738-14927-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790722213368738-14927-0/krb5kdc/kudu_127.14.147.254.keytab
--principal=kudu/127.14.147.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.147.254:37205
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790722213368738-14927-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.14.147.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27141
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790722213368738-14927-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 21:58:54 UTC on ccycloud.quasar-xguoti.root.comops.site
build id 4681390
I20260929 22:52:38.843467 27141 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 22:52:38.843693 27141 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 22:52:38.848747 27146 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 22:52:38.848845 27141 server_base.cc:1034] running on GCE node
W20260929 22:52:38.848752 27147 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 22:52:38.849025 27149 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 22:52:38.849355 27141 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 22:52:38.849676 27141 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 22:52:38.850821 27141 hybrid_clock.cc:648] HybridClock initialized: now 1790722358850799 us; error 38 us; skew 500 ppm
Sep 29 22:52:38 dist-test-slave-98mk krb5kdc[26697](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790722358, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.147.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 22:52:38.853521 27141 init.cc:377] Logged in from keytab as kudu/127.14.147.254@KRBTEST.COM (short username kudu)
I20260929 22:52:38.854015 27141 webserver.cc:460] Webserver started at http://127.14.147.254:38013/ using document root <none> and password file <none>
I20260929 22:52:38.854221 27141 fs_manager.cc:359] Metadata directory not provided
I20260929 22:52:38.854269 27141 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 22:52:38.854395 27141 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 29 22:52:38 dist-test-slave-98mk krb5kdc[26697](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790722358, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.147.254@KRBTEST.COM for HTTP/127.14.147.212@KRBTEST.COM
I20260929 22:52:38.908531 27141 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790722213368738-14927-0/minicluster-data/master-0/data/instance:
uuid: "31ac56a028ae4c99bcfe0cde4b1efc91"
format_stamp: "Formatted at 2026-09-29 22:52:38 on dist-test-slave-98mk"
server_key: "3d409f837d75ae0f65292038860e6ebe"
server_key_iv: "0f03ebdb06c4f1f32301aded3561c7ef"
server_key_version: "kuduclusterkey@0"
I20260929 22:52:38.909003 27141 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790722213368738-14927-0/minicluster-data/master-0/wal/instance:
uuid: "31ac56a028ae4c99bcfe0cde4b1efc91"
format_stamp: "Formatted at 2026-09-29 22:52:38 on dist-test-slave-98mk"
server_key: "3d409f837d75ae0f65292038860e6ebe"
server_key_iv: "0f03ebdb06c4f1f32301aded3561c7ef"
server_key_version: "kuduclusterkey@0"
I20260929 22:52:38.910755 27141 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.002s sys 0.000s
I20260929 22:52:38.922925 27156 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 22:52:38.923209 27141 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260929 22:52:38.923316 27141 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790722213368738-14927-0/minicluster-data/master-0/data,/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790722213368738-14927-0/minicluster-data/master-0/wal
uuid: "31ac56a028ae4c99bcfe0cde4b1efc91"
format_stamp: "Formatted at 2026-09-29 22:52:38 on dist-test-slave-98mk"
server_key: "3d409f837d75ae0f65292038860e6ebe"
server_key_iv: "0f03ebdb06c4f1f32301aded3561c7ef"
server_key_version: "kuduclusterkey@0"
I20260929 22:52:38.923444 27141 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790722213368738-14927-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790722213368738-14927-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790722213368738-14927-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 22:52:38.939972 27141 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 22:52:38.941859 27141 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 22:52:38.942008 27141 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 22:52:38.946584 27141 rpc_server.cc:307] RPC server started. Bound to: 127.14.147.254:37205
I20260929 22:52:38.946610 27208 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.147.254:37205 every 8 connection(s)
I20260929 22:52:38.946990 27141 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790722213368738-14927-0/minicluster-data/master-0/data/info.pb
I20260929 22:52:38.947839 27209 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 22:52:38.951126 27209 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 31ac56a028ae4c99bcfe0cde4b1efc91: Bootstrap starting.
I20260929 22:52:38.951604 14927 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhWOCLk/build/release/bin/kudu as pid 27141
I20260929 22:52:38.951689 14927 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790722213368738-14927-0/minicluster-data/master-0/wal/instance
I20260929 22:52:38.951857 14927 external_mini_cluster.cc:1428] Setting key 176ab5a9575f84254f030a12ac244494
I20260929 22:52:38.951889 27209 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 31ac56a028ae4c99bcfe0cde4b1efc91: Neither blocks nor log segments found. Creating new log.
I20260929 22:52:38.952189 27209 log.cc:824] T 00000000000000000000000000000000 P 31ac56a028ae4c99bcfe0cde4b1efc91: Log is configured to *not* fsync() on all Append() calls
I20260929 22:52:38.954697 27209 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 31ac56a028ae4c99bcfe0cde4b1efc91: No bootstrap required, opened a new log
I20260929 22:52:38.957790 27209 raft_consensus.cc:348] T 00000000000000000000000000000000 P 31ac56a028ae4c99bcfe0cde4b1efc91 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "31ac56a028ae4c99bcfe0cde4b1efc91" member_type: VOTER last_known_addr { host: "127.14.147.254" port: 37205 } }
I20260929 22:52:38.957970 27209 raft_consensus.cc:374] T 00000000000000000000000000000000 P 31ac56a028ae4c99bcfe0cde4b1efc91 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 22:52:38.958004 27209 raft_consensus.cc:729] T 00000000000000000000000000000000 P 31ac56a028ae4c99bcfe0cde4b1efc91 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 31ac56a028ae4c99bcfe0cde4b1efc91, State: Initialized, Role: FOLLOWER
I20260929 22:52:38.958153 27209 consensus_queue.cc:260] T 00000000000000000000000000000000 P 31ac56a028ae4c99bcfe0cde4b1efc91 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "31ac56a028ae4c99bcfe0cde4b1efc91" member_type: VOTER last_known_addr { host: "127.14.147.254" port: 37205 } }
I20260929 22:52:38.958235 27209 raft_consensus.cc:388] T 00000000000000000000000000000000 P 31ac56a028ae4c99bcfe0cde4b1efc91 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 22:52:38.958267 27209 raft_consensus.cc:482] T 00000000000000000000000000000000 P 31ac56a028ae4c99bcfe0cde4b1efc91 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 22:52:38.958351 27209 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 31ac56a028ae4c99bcfe0cde4b1efc91 [term 0 FOLLOWER]: Advancing to term 1
I20260929 22:52:38.959182 27209 raft_consensus.cc:504] T 00000000000000000000000000000000 P 31ac56a028ae4c99bcfe0cde4b1efc91 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "31ac56a028ae4c99bcfe0cde4b1efc91" member_type: VOTER last_known_addr { host: "127.14.147.254" port: 37205 } }
I20260929 22:52:38.959342 27209 leader_election.cc:302] T 00000000000000000000000000000000 P 31ac56a028ae4c99bcfe0cde4b1efc91 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 31ac56a028ae4c99bcfe0cde4b1efc91; no voters:
I20260929 22:52:38.959534 27209 leader_election.cc:288] T 00000000000000000000000000000000 P 31ac56a028ae4c99bcfe0cde4b1efc91 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 22:52:38.959563 27214 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 31ac56a028ae4c99bcfe0cde4b1efc91 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 22:52:38.959800 27214 raft_consensus.cc:686] T 00000000000000000000000000000000 P 31ac56a028ae4c99bcfe0cde4b1efc91 [term 1 LEADER]: Becoming Leader. State: Replica: 31ac56a028ae4c99bcfe0cde4b1efc91, State: Running, Role: LEADER
Sep 29 22:52:38 dist-test-slave-98mk krb5kdc[26697](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790722358, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.147.254@KRBTEST.COM
I20260929 22:52:38.959903 27209 sys_catalog.cc:564] T 00000000000000000000000000000000 P 31ac56a028ae4c99bcfe0cde4b1efc91 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 22:52:38.959954 27214 consensus_queue.cc:237] T 00000000000000000000000000000000 P 31ac56a028ae4c99bcfe0cde4b1efc91 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "31ac56a028ae4c99bcfe0cde4b1efc91" member_type: VOTER last_known_addr { host: "127.14.147.254" port: 37205 } }
I20260929 22:52:38.960886 27216 sys_catalog.cc:455] T 00000000000000000000000000000000 P 31ac56a028ae4c99bcfe0cde4b1efc91 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "31ac56a028ae4c99bcfe0cde4b1efc91" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "31ac56a028ae4c99bcfe0cde4b1efc91" member_type: VOTER last_known_addr { host: "127.14.147.254" port: 37205 } } }
I20260929 22:52:38.960970 27216 sys_catalog.cc:458] T 00000000000000000000000000000000 P 31ac56a028ae4c99bcfe0cde4b1efc91 [sys.catalog]: This master's current role is: LEADER
I20260929 22:52:38.961362 27209 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790722213368738-14927-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260929 22:52:38.961367 27215 sys_catalog.cc:455] T 00000000000000000000000000000000 P 31ac56a028ae4c99bcfe0cde4b1efc91 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 31ac56a028ae4c99bcfe0cde4b1efc91. Latest consensus state: current_term: 1 leader_uuid: "31ac56a028ae4c99bcfe0cde4b1efc91" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "31ac56a028ae4c99bcfe0cde4b1efc91" member_type: VOTER last_known_addr { host: "127.14.147.254" port: 37205 } } }
I20260929 22:52:38.961532 27215 sys_catalog.cc:458] T 00000000000000000000000000000000 P 31ac56a028ae4c99bcfe0cde4b1efc91 [sys.catalog]: This master's current role is: LEADER
I20260929 22:52:38.961716 27228 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 22:52:38.966084 27228 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 22:52:38.967401 27212 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:52:38.952325 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41658 (local address 127.14.147.254:37205)
0929 22:52:38.952529 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0929 22:52:38.952538 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 22:52:38.952572 (+ 34us) server_negotiation.cc:407] Connection header received
0929 22:52:38.953166 (+ 594us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 22:52:38.953176 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 22:52:38.953254 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 22:52:38.953425 (+ 171us) server_negotiation.cc:226] Negotiated authn=SASL
0929 22:52:38.953980 (+ 555us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:52:38.955446 (+ 1466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:52:38.956824 (+ 1378us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:52:38.957468 (+ 644us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:52:38.957588 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 22:52:38.960444 (+ 2856us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 22:52:38.960449 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 22:52:38.960450 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 22:52:38.960467 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0929 22:52:38.964905 (+ 4438us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 22:52:38.965662 (+ 757us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 22:52:38.965667 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 22:52:38.965668 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 22:52:38.965836 (+ 168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 22:52:38.966054 (+ 218us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 22:52:38.966055 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 22:52:38.966055 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 22:52:38.966496 (+ 441us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 22:52:38.966728 (+ 232us) server_negotiation.cc:1035] Waiting for connection context
0929 22:52:38.966797 (+ 69us) server_negotiation.cc:299] Negotiation successful
0929 22:52:38.967145 (+ 348us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":155,"thread_start_us":74,"threads_started":1}
I20260929 22:52:38.970214 27228 catalog_manager.cc:1347] Generated new cluster ID: fb7d2c791dad49a99bc01d9324f688be
I20260929 22:52:38.970283 27228 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 22:52:38.977880 27228 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 22:52:38.978710 27228 catalog_manager.cc:1504] Loading token signing keys...
I20260929 22:52:38.985163 27228 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 31ac56a028ae4c99bcfe0cde4b1efc91: Generated new TSK 0
I20260929 22:52:38.985517 27228 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.14.147.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.147.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.147.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790722213368738-14927-0/krb5kdc/kudu_127.14.147.193.keytab.
Entry for principal kudu/127.14.147.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790722213368738-14927-0/krb5kdc/kudu_127.14.147.193.keytab.
I20260929 22:52:39.733839 14927 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhWOCLk/build/release/bin/kudu
/tmp/dist-test-taskhWOCLk/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790722213368738-14927-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790722213368738-14927-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790722213368738-14927-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790722213368738-14927-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.14.147.212:46007/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.147.193:0
--local_ip_for_outbound_sockets=127.14.147.193
--webserver_interface=127.14.147.193
--webserver_port=0
--tserver_master_addrs=127.14.147.254:37205
--keytab_file=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790722213368738-14927-0/krb5kdc/kudu_127.14.147.193.keytab
--principal=kudu/127.14.147.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.147.212:33235
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790722213368738-14927-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790722213368738-14927-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790722213368738-14927-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 22:52:39.818611 27254 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 22:52:39.818822 27254 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 22:52:39.818857 27254 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 22:52:39.820143 27254 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 22:52:39.820194 27254 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 22:52:39.820231 27254 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.147.193
I20260929 22:52:39.821830 27254 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.147.212:33235
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790722213368738-14927-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790722213368738-14927-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.14.147.212:46007/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790722213368738-14927-0/krb5kdc/kudu_127.14.147.193.keytab
--principal=kudu/127.14.147.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.147.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790722213368738-14927-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.14.147.193
--webserver_port=0
--tserver_master_addrs=127.14.147.254:37205
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27254
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.147.193
--log_dir=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790722213368738-14927-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 21:58:54 UTC on ccycloud.quasar-xguoti.root.comops.site
build id 4681390
I20260929 22:52:39.822072 27254 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 22:52:39.822310 27254 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 22:52:39.827037 27262 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 22:52:39.827037 27260 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 22:52:39.827399 27259 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 22:52:39.827862 27254 server_base.cc:1034] running on GCE node
I20260929 22:52:39.828063 27254 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 22:52:39.828332 27254 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 22:52:39.829497 27254 hybrid_clock.cc:648] HybridClock initialized: now 1790722359829480 us; error 35 us; skew 500 ppm
Sep 29 22:52:39 dist-test-slave-98mk krb5kdc[26697](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790722359, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.147.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 22:52:39.831995 27254 init.cc:377] Logged in from keytab as kudu/127.14.147.193@KRBTEST.COM (short username kudu)
I20260929 22:52:39.832438 27254 webserver.cc:460] Webserver started at http://127.14.147.193:41789/ using document root <none> and password file <none>
I20260929 22:52:39.832648 27254 fs_manager.cc:359] Metadata directory not provided
I20260929 22:52:39.832700 27254 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 22:52:39.832819 27254 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 29 22:52:39 dist-test-slave-98mk krb5kdc[26697](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790722359, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.147.193@KRBTEST.COM for HTTP/127.14.147.212@KRBTEST.COM
I20260929 22:52:39.852160 27254 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790722213368738-14927-0/minicluster-data/ts-0/data/instance:
uuid: "738ca65869ba46809f58b052437e15e2"
format_stamp: "Formatted at 2026-09-29 22:52:39 on dist-test-slave-98mk"
server_key: "745a5cf4355e16629eb67479bdc86c7f"
server_key_iv: "cbf341961497f263dbb730b5d5dff745"
server_key_version: "kuduclusterkey@0"
I20260929 22:52:39.852804 27254 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790722213368738-14927-0/minicluster-data/ts-0/wal/instance:
uuid: "738ca65869ba46809f58b052437e15e2"
format_stamp: "Formatted at 2026-09-29 22:52:39 on dist-test-slave-98mk"
server_key: "745a5cf4355e16629eb67479bdc86c7f"
server_key_iv: "cbf341961497f263dbb730b5d5dff745"
server_key_version: "kuduclusterkey@0"
I20260929 22:52:39.855212 27254 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.001s sys 0.000s
I20260929 22:52:39.879637 27269 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 22:52:39.879900 27254 fs_manager.cc:725] Time spent opening block manager: real 0.006s user 0.001s sys 0.000s
I20260929 22:52:39.880004 27254 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790722213368738-14927-0/minicluster-data/ts-0/data,/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790722213368738-14927-0/minicluster-data/ts-0/wal
uuid: "738ca65869ba46809f58b052437e15e2"
format_stamp: "Formatted at 2026-09-29 22:52:39 on dist-test-slave-98mk"
server_key: "745a5cf4355e16629eb67479bdc86c7f"
server_key_iv: "cbf341961497f263dbb730b5d5dff745"
server_key_version: "kuduclusterkey@0"
I20260929 22:52:39.880112 27254 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790722213368738-14927-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790722213368738-14927-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790722213368738-14927-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 22:52:39.894636 27254 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 22:52:39.898123 27254 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 22:52:39.898267 27254 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 22:52:39.898495 27254 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 22:52:39.898849 27254 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 22:52:39.898893 27254 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 22:52:39.898926 27254 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 22:52:39.898948 27254 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 22:52:39.906740 27254 rpc_server.cc:307] RPC server started. Bound to: 127.14.147.193:44211
I20260929 22:52:39.907305 27254 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790722213368738-14927-0/minicluster-data/ts-0/data/info.pb
I20260929 22:52:39.907471 27384 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.147.193:44211 every 8 connection(s)
I20260929 22:52:39.911149 14927 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhWOCLk/build/release/bin/kudu as pid 27254
I20260929 22:52:39.911247 14927 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790722213368738-14927-0/minicluster-data/ts-0/wal/instance
I20260929 22:52:39.911444 14927 external_mini_cluster.cc:1428] Setting key 5e7076de1f743c48b49c5e5397e24655
Sep 29 22:52:39 dist-test-slave-98mk krb5kdc[26697](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790722359, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.147.193@KRBTEST.COM for kudu/127.14.147.254@KRBTEST.COM
I20260929 22:52:39.919855 27387 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:52:39.908219 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.147.254:37205 (local address 127.14.147.193:39699)
0929 22:52:39.908378 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0929 22:52:39.908421 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0929 22:52:39.908895 (+ 474us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 22:52:39.909262 (+ 367us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 22:52:39.909266 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 22:52:39.909457 (+ 191us) client_negotiation.cc:189] Negotiated authn=SASL
0929 22:52:39.909647 (+ 190us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 22:52:39.909651 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 22:52:39.910345 (+ 694us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 22:52:39.910346 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 22:52:39.911276 (+ 930us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 22:52:39.911280 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 22:52:39.911975 (+ 695us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 22:52:39.911977 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 22:52:39.913611 (+ 1634us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 22:52:39.914225 (+ 614us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 22:52:39.914235 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0929 22:52:39.915939 (+ 1704us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 22:52:39.917891 (+ 1952us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 22:52:39.917899 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 22:52:39.917901 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 22:52:39.918186 (+ 285us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 22:52:39.918400 (+ 214us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 22:52:39.918400 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 22:52:39.918401 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 22:52:39.918530 (+ 129us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 22:52:39.918833 (+ 303us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 22:52:39.918839 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 22:52:39.919006 (+ 167us) client_negotiation.cc:769] Sending connection context
0929 22:52:39.919091 (+ 85us) client_negotiation.cc:240] Negotiation successful
0929 22:52:39.919247 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":73,"threads_started":1}
I20260929 22:52:39.920360 27388 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:52:39.908257 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.147.193:39699 (local address 127.14.147.254:37205)
0929 22:52:39.909033 (+ 776us) server_negotiation.cc:206] Beginning negotiation
0929 22:52:39.909035 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 22:52:39.909042 (+ 7us) server_negotiation.cc:407] Connection header received
0929 22:52:39.909063 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 22:52:39.909063 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 22:52:39.909073 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 22:52:39.909126 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0929 22:52:39.909741 (+ 615us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:52:39.910257 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:52:39.911376 (+ 1119us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:52:39.911703 (+ 327us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:52:39.911770 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 22:52:39.916063 (+ 4293us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 22:52:39.916065 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 22:52:39.916065 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 22:52:39.916073 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0929 22:52:39.917785 (+ 1712us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 22:52:39.918272 (+ 487us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 22:52:39.918273 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 22:52:39.918273 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 22:52:39.918336 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 22:52:39.918604 (+ 268us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 22:52:39.918604 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 22:52:39.918605 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 22:52:39.918760 (+ 155us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 22:52:39.920125 (+ 1365us) server_negotiation.cc:1035] Waiting for connection context
0929 22:52:39.920168 (+ 43us) server_negotiation.cc:299] Negotiation successful
0929 22:52:39.920257 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":748,"thread_start_us":85,"threads_started":1}
I20260929 22:52:39.920899 27385 heartbeater.cc:347] Connected to a master server at 127.14.147.254:37205
I20260929 22:52:39.920990 27385 heartbeater.cc:464] Registering TS with master...
I20260929 22:52:39.921168 27385 heartbeater.cc:511] Master 127.14.147.254:37205 requested a full tablet report, sending...
I20260929 22:52:39.921808 27173 ts_manager.cc:194] Registered new tserver with Master: 738ca65869ba46809f58b052437e15e2 (127.14.147.193:44211)
I20260929 22:52:39.922596 27173 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.147.193@KRBTEST.COM'} at 127.14.147.193:39699
WARNING: no policy specified for kudu/127.14.147.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.147.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.147.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790722213368738-14927-0/krb5kdc/kudu_127.14.147.194.keytab.
Entry for principal kudu/127.14.147.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790722213368738-14927-0/krb5kdc/kudu_127.14.147.194.keytab.
I20260929 22:52:39.946573 14927 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhWOCLk/build/release/bin/kudu
/tmp/dist-test-taskhWOCLk/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790722213368738-14927-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790722213368738-14927-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790722213368738-14927-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790722213368738-14927-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.14.147.212:46007/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.147.194:0
--local_ip_for_outbound_sockets=127.14.147.194
--webserver_interface=127.14.147.194
--webserver_port=0
--tserver_master_addrs=127.14.147.254:37205
--keytab_file=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790722213368738-14927-0/krb5kdc/kudu_127.14.147.194.keytab
--principal=kudu/127.14.147.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.147.212:33235
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790722213368738-14927-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790722213368738-14927-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790722213368738-14927-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 22:52:40.068001 27391 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 22:52:40.068318 27391 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 22:52:40.068361 27391 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 22:52:40.070286 27391 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 22:52:40.070354 27391 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 22:52:40.070396 27391 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.147.194
I20260929 22:52:40.072918 27391 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.147.212:33235
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790722213368738-14927-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790722213368738-14927-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.14.147.212:46007/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790722213368738-14927-0/krb5kdc/kudu_127.14.147.194.keytab
--principal=kudu/127.14.147.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.147.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790722213368738-14927-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.14.147.194
--webserver_port=0
--tserver_master_addrs=127.14.147.254:37205
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27391
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.147.194
--log_dir=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790722213368738-14927-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 21:58:54 UTC on ccycloud.quasar-xguoti.root.comops.site
build id 4681390
I20260929 22:52:40.073213 27391 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 22:52:40.073555 27391 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 22:52:40.079988 27398 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 22:52:40.080003 27399 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 22:52:40.080744 27401 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 22:52:40.081327 27391 server_base.cc:1034] running on GCE node
I20260929 22:52:40.081530 27391 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 22:52:40.081777 27391 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 22:52:40.082952 27391 hybrid_clock.cc:648] HybridClock initialized: now 1790722360082931 us; error 34 us; skew 500 ppm
Sep 29 22:52:40 dist-test-slave-98mk krb5kdc[26697](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790722360, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.147.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 22:52:40.085598 27391 init.cc:377] Logged in from keytab as kudu/127.14.147.194@KRBTEST.COM (short username kudu)
I20260929 22:52:40.086100 27391 webserver.cc:460] Webserver started at http://127.14.147.194:36753/ using document root <none> and password file <none>
I20260929 22:52:40.086341 27391 fs_manager.cc:359] Metadata directory not provided
I20260929 22:52:40.086390 27391 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 22:52:40.086522 27391 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 29 22:52:40 dist-test-slave-98mk krb5kdc[26697](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790722360, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.147.194@KRBTEST.COM for HTTP/127.14.147.212@KRBTEST.COM
Sep 29 22:52:40 dist-test-slave-98mk krb5kdc[26697](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790722360, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.147.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 22:52:40.100098 27391 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790722213368738-14927-0/minicluster-data/ts-1/data/instance:
uuid: "fcf7eed1fd374116b677881d7eaadbd4"
format_stamp: "Formatted at 2026-09-29 22:52:40 on dist-test-slave-98mk"
server_key: "f88afea91e8cafe052ad8b6ee0130095"
server_key_iv: "41408663550f21bf2eead47819242edf"
server_key_version: "kuduclusterkey@0"
I20260929 22:52:40.100559 27391 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790722213368738-14927-0/minicluster-data/ts-1/wal/instance:
uuid: "fcf7eed1fd374116b677881d7eaadbd4"
format_stamp: "Formatted at 2026-09-29 22:52:40 on dist-test-slave-98mk"
server_key: "f88afea91e8cafe052ad8b6ee0130095"
server_key_iv: "41408663550f21bf2eead47819242edf"
server_key_version: "kuduclusterkey@0"
I20260929 22:52:40.102325 27391 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.003s sys 0.000s
I20260929 22:52:40.126785 27408 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 22:52:40.127081 27391 fs_manager.cc:725] Time spent opening block manager: real 0.012s user 0.001s sys 0.000s
I20260929 22:52:40.127238 27391 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790722213368738-14927-0/minicluster-data/ts-1/data,/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790722213368738-14927-0/minicluster-data/ts-1/wal
uuid: "fcf7eed1fd374116b677881d7eaadbd4"
format_stamp: "Formatted at 2026-09-29 22:52:40 on dist-test-slave-98mk"
server_key: "f88afea91e8cafe052ad8b6ee0130095"
server_key_iv: "41408663550f21bf2eead47819242edf"
server_key_version: "kuduclusterkey@0"
I20260929 22:52:40.127396 27391 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790722213368738-14927-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790722213368738-14927-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790722213368738-14927-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 22:52:40.143788 27391 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 22:52:40.146334 27391 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 22:52:40.146528 27391 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 22:52:40.146781 27391 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 22:52:40.147192 27391 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 22:52:40.147265 27391 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 22:52:40.147326 27391 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 22:52:40.147372 27391 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 22:52:40.153699 27391 rpc_server.cc:307] RPC server started. Bound to: 127.14.147.194:38247
I20260929 22:52:40.153831 27521 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.147.194:38247 every 8 connection(s)
I20260929 22:52:40.154201 27391 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790722213368738-14927-0/minicluster-data/ts-1/data/info.pb
I20260929 22:52:40.155217 14927 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhWOCLk/build/release/bin/kudu as pid 27391
I20260929 22:52:40.155304 14927 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790722213368738-14927-0/minicluster-data/ts-1/wal/instance
I20260929 22:52:40.155512 14927 external_mini_cluster.cc:1428] Setting key d2a0d48334a685ca7887a144ca392abf
Sep 29 22:52:40 dist-test-slave-98mk krb5kdc[26697](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790722360, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.147.194@KRBTEST.COM for kudu/127.14.147.254@KRBTEST.COM
I20260929 22:52:40.169673 27388 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:52:40.154962 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.147.194:42941 (local address 127.14.147.254:37205)
0929 22:52:40.155031 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0929 22:52:40.155033 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 22:52:40.155311 (+ 278us) server_negotiation.cc:407] Connection header received
0929 22:52:40.156231 (+ 920us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 22:52:40.156232 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 22:52:40.156244 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 22:52:40.156270 (+ 26us) server_negotiation.cc:226] Negotiated authn=SASL
0929 22:52:40.157117 (+ 847us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:52:40.158362 (+ 1245us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:52:40.160473 (+ 2111us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:52:40.160937 (+ 464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:52:40.161050 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 22:52:40.165906 (+ 4856us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 22:52:40.165908 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 22:52:40.165908 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 22:52:40.165918 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0929 22:52:40.167886 (+ 1968us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 22:52:40.168484 (+ 598us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 22:52:40.168485 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 22:52:40.168486 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 22:52:40.168555 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 22:52:40.168868 (+ 313us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 22:52:40.168869 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 22:52:40.168869 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 22:52:40.169020 (+ 151us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 22:52:40.169082 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0929 22:52:40.169439 (+ 357us) server_negotiation.cc:299] Negotiation successful
0929 22:52:40.169542 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260929 22:52:40.169811 27524 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:52:40.154931 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.147.254:37205 (local address 127.14.147.194:42941)
0929 22:52:40.155194 (+ 263us) negotiation.cc:107] Waiting for socket to connect
0929 22:52:40.155236 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0929 22:52:40.156069 (+ 833us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 22:52:40.156439 (+ 370us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 22:52:40.156443 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 22:52:40.156741 (+ 298us) client_negotiation.cc:189] Negotiated authn=SASL
0929 22:52:40.157010 (+ 269us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 22:52:40.157016 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 22:52:40.158578 (+ 1562us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 22:52:40.158578 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 22:52:40.160360 (+ 1782us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 22:52:40.160363 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 22:52:40.161054 (+ 691us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 22:52:40.161055 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 22:52:40.161197 (+ 142us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 22:52:40.163597 (+ 2400us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 22:52:40.163609 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0929 22:52:40.165758 (+ 2149us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 22:52:40.167998 (+ 2240us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 22:52:40.168009 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 22:52:40.168012 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 22:52:40.168370 (+ 358us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 22:52:40.168630 (+ 260us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 22:52:40.168631 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 22:52:40.168631 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 22:52:40.168788 (+ 157us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 22:52:40.169085 (+ 297us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 22:52:40.169091 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 22:52:40.169280 (+ 189us) client_negotiation.cc:769] Sending connection context
0929 22:52:40.169390 (+ 110us) client_negotiation.cc:240] Negotiation successful
0929 22:52:40.169561 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":226,"thread_start_us":69,"threads_started":1}
I20260929 22:52:40.170671 27522 heartbeater.cc:347] Connected to a master server at 127.14.147.254:37205
I20260929 22:52:40.170784 27522 heartbeater.cc:464] Registering TS with master...
I20260929 22:52:40.171031 27522 heartbeater.cc:511] Master 127.14.147.254:37205 requested a full tablet report, sending...
I20260929 22:52:40.171676 27173 ts_manager.cc:194] Registered new tserver with Master: fcf7eed1fd374116b677881d7eaadbd4 (127.14.147.194:38247)
I20260929 22:52:40.172148 27173 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.147.194@KRBTEST.COM'} at 127.14.147.194:42941
WARNING: no policy specified for kudu/127.14.147.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.147.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.147.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790722213368738-14927-0/krb5kdc/kudu_127.14.147.195.keytab.
Entry for principal kudu/127.14.147.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790722213368738-14927-0/krb5kdc/kudu_127.14.147.195.keytab.
I20260929 22:52:40.205667 14927 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhWOCLk/build/release/bin/kudu
/tmp/dist-test-taskhWOCLk/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790722213368738-14927-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790722213368738-14927-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790722213368738-14927-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790722213368738-14927-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.14.147.212:46007/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.147.195:0
--local_ip_for_outbound_sockets=127.14.147.195
--webserver_interface=127.14.147.195
--webserver_port=0
--tserver_master_addrs=127.14.147.254:37205
--keytab_file=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790722213368738-14927-0/krb5kdc/kudu_127.14.147.195.keytab
--principal=kudu/127.14.147.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.147.212:33235
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790722213368738-14927-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790722213368738-14927-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790722213368738-14927-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 22:52:40.326980 27528 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 22:52:40.327347 27528 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 22:52:40.327489 27528 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 22:52:40.329459 27528 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 22:52:40.329605 27528 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 22:52:40.329699 27528 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.147.195
I20260929 22:52:40.332526 27528 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.147.212:33235
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790722213368738-14927-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790722213368738-14927-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.14.147.212:46007/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790722213368738-14927-0/krb5kdc/kudu_127.14.147.195.keytab
--principal=kudu/127.14.147.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.147.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790722213368738-14927-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.14.147.195
--webserver_port=0
--tserver_master_addrs=127.14.147.254:37205
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27528
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.147.195
--log_dir=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790722213368738-14927-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 21:58:54 UTC on ccycloud.quasar-xguoti.root.comops.site
build id 4681390
I20260929 22:52:40.333073 27528 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 22:52:40.333403 27528 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 22:52:40.340216 27534 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 22:52:40.341521 27528 server_base.cc:1034] running on GCE node
W20260929 22:52:40.343586 27533 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 22:52:40.344636 27536 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 22:52:40.344991 27528 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 22:52:40.345351 27528 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 22:52:40.346552 27528 hybrid_clock.cc:648] HybridClock initialized: now 1790722360346520 us; error 47 us; skew 500 ppm
Sep 29 22:52:40 dist-test-slave-98mk krb5kdc[26697](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790722360, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.147.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 22:52:40.349980 27528 init.cc:377] Logged in from keytab as kudu/127.14.147.195@KRBTEST.COM (short username kudu)
I20260929 22:52:40.350392 27528 webserver.cc:460] Webserver started at http://127.14.147.195:36211/ using document root <none> and password file <none>
I20260929 22:52:40.350590 27528 fs_manager.cc:359] Metadata directory not provided
I20260929 22:52:40.350622 27528 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 22:52:40.350737 27528 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 29 22:52:40 dist-test-slave-98mk krb5kdc[26697](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790722360, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.147.195@KRBTEST.COM for HTTP/127.14.147.212@KRBTEST.COM
I20260929 22:52:40.373313 27528 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790722213368738-14927-0/minicluster-data/ts-2/data/instance:
uuid: "a143941d3e8349a2941b79eb89915ba4"
format_stamp: "Formatted at 2026-09-29 22:52:40 on dist-test-slave-98mk"
server_key: "409f10ac47621fa46fd3f3f7044c847a"
server_key_iv: "20e9bcb5f455c8bf4f89d84978bf6fad"
server_key_version: "kuduclusterkey@0"
I20260929 22:52:40.373783 27528 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790722213368738-14927-0/minicluster-data/ts-2/wal/instance:
uuid: "a143941d3e8349a2941b79eb89915ba4"
format_stamp: "Formatted at 2026-09-29 22:52:40 on dist-test-slave-98mk"
server_key: "409f10ac47621fa46fd3f3f7044c847a"
server_key_iv: "20e9bcb5f455c8bf4f89d84978bf6fad"
server_key_version: "kuduclusterkey@0"
I20260929 22:52:40.383898 27528 fs_manager.cc:691] Time spent creating directory manager: real 0.010s user 0.002s sys 0.000s
I20260929 22:52:40.399250 27543 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 22:52:40.399467 27528 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260929 22:52:40.399564 27528 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790722213368738-14927-0/minicluster-data/ts-2/data,/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790722213368738-14927-0/minicluster-data/ts-2/wal
uuid: "a143941d3e8349a2941b79eb89915ba4"
format_stamp: "Formatted at 2026-09-29 22:52:40 on dist-test-slave-98mk"
server_key: "409f10ac47621fa46fd3f3f7044c847a"
server_key_iv: "20e9bcb5f455c8bf4f89d84978bf6fad"
server_key_version: "kuduclusterkey@0"
I20260929 22:52:40.399657 27528 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790722213368738-14927-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790722213368738-14927-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790722213368738-14927-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 22:52:40.410002 27528 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 22:52:40.412911 27528 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 22:52:40.413151 27528 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 22:52:40.413479 27528 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 22:52:40.413894 27528 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 22:52:40.413982 27528 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 22:52:40.414063 27528 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 22:52:40.414119 27528 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 22:52:40.420678 27528 rpc_server.cc:307] RPC server started. Bound to: 127.14.147.195:41647
I20260929 22:52:40.420751 27656 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.147.195:41647 every 8 connection(s)
I20260929 22:52:40.421124 27528 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790722213368738-14927-0/minicluster-data/ts-2/data/info.pb
I20260929 22:52:40.422793 14927 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhWOCLk/build/release/bin/kudu as pid 27528
I20260929 22:52:40.422883 14927 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790722213368738-14927-0/minicluster-data/ts-2/wal/instance
I20260929 22:52:40.423048 14927 external_mini_cluster.cc:1428] Setting key 6ab53a866d48358e45f9d9dd2e66ae50
Sep 29 22:52:40 dist-test-slave-98mk krb5kdc[26697](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790722360, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.147.195@KRBTEST.COM for kudu/127.14.147.254@KRBTEST.COM
I20260929 22:52:40.433354 27388 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:52:40.421886 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.147.195:52341 (local address 127.14.147.254:37205)
0929 22:52:40.421935 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0929 22:52:40.421937 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 22:52:40.422157 (+ 220us) server_negotiation.cc:407] Connection header received
0929 22:52:40.423211 (+ 1054us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 22:52:40.423212 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 22:52:40.423226 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 22:52:40.423282 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0929 22:52:40.424123 (+ 841us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:52:40.424649 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:52:40.425617 (+ 968us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:52:40.425879 (+ 262us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:52:40.425938 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 22:52:40.429920 (+ 3982us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 22:52:40.429922 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 22:52:40.429922 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 22:52:40.429931 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0929 22:52:40.431596 (+ 1665us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 22:52:40.432165 (+ 569us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 22:52:40.432166 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 22:52:40.432167 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 22:52:40.432225 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 22:52:40.432501 (+ 276us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 22:52:40.432502 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 22:52:40.432502 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 22:52:40.432646 (+ 144us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 22:52:40.432687 (+ 41us) server_negotiation.cc:1035] Waiting for connection context
0929 22:52:40.433156 (+ 469us) server_negotiation.cc:299] Negotiation successful
0929 22:52:40.433248 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260929 22:52:40.433905 27659 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:52:40.421876 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.147.254:37205 (local address 127.14.147.195:52341)
0929 22:52:40.422027 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0929 22:52:40.422068 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0929 22:52:40.422583 (+ 515us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 22:52:40.423451 (+ 868us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 22:52:40.423456 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 22:52:40.423759 (+ 303us) client_negotiation.cc:189] Negotiated authn=SASL
0929 22:52:40.424025 (+ 266us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 22:52:40.424031 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 22:52:40.424733 (+ 702us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 22:52:40.424734 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 22:52:40.425521 (+ 787us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 22:52:40.425523 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 22:52:40.425992 (+ 469us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 22:52:40.425993 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 22:52:40.426107 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 22:52:40.428223 (+ 2116us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 22:52:40.428235 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0929 22:52:40.429792 (+ 1557us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 22:52:40.431722 (+ 1930us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 22:52:40.431730 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 22:52:40.431733 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 22:52:40.432082 (+ 349us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 22:52:40.432288 (+ 206us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 22:52:40.432289 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 22:52:40.432289 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 22:52:40.432431 (+ 142us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 22:52:40.432777 (+ 346us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 22:52:40.432784 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 22:52:40.432960 (+ 176us) client_negotiation.cc:769] Sending connection context
0929 22:52:40.433494 (+ 534us) client_negotiation.cc:240] Negotiation successful
0929 22:52:40.433669 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":83,"threads_started":1}
I20260929 22:52:40.435847 27657 heartbeater.cc:347] Connected to a master server at 127.14.147.254:37205
I20260929 22:52:40.435954 27657 heartbeater.cc:464] Registering TS with master...
I20260929 22:52:40.436149 27657 heartbeater.cc:511] Master 127.14.147.254:37205 requested a full tablet report, sending...
I20260929 22:52:40.436523 27173 ts_manager.cc:194] Registered new tserver with Master: a143941d3e8349a2941b79eb89915ba4 (127.14.147.195:41647)
I20260929 22:52:40.436862 14927 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 22:52:40.436986 27173 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.147.195@KRBTEST.COM'} at 127.14.147.195:52341
I20260929 22:52:40.437621 14927 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhWOCLk/build/release/bin/kudu with pid 27254
I20260929 22:52:40.443007 14927 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhWOCLk/build/release/bin/kudu with pid 27391
I20260929 22:52:40.451999 14927 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhWOCLk/build/release/bin/kudu with pid 27528
I20260929 22:52:40.458088 14927 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhWOCLk/build/release/bin/kudu with pid 27141
I20260929 22:52:40.462991 14927 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhWOCLk/build/release/bin/kudu
/tmp/dist-test-taskhWOCLk/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790722213368738-14927-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790722213368738-14927-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790722213368738-14927-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790722213368738-14927-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.147.254:37205
--webserver_interface=127.14.147.254
--webserver_port=38013
--keytab_file=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790722213368738-14927-0/krb5kdc/kudu_127.14.147.254.keytab
--principal=kudu/127.14.147.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.147.212:33235
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.147.254:37205
--ranger_config_path=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790722213368738-14927-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.14.147.212:46007/kms
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.14.147.212:46007/kms with env {KRB5CCNAME=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790722213368738-14927-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790722213368738-14927-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790722213368738-14927-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 22:52:40.544003 27660 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 22:52:40.544236 27660 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 22:52:40.544267 27660 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 22:52:40.545578 27660 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 22:52:40.545634 27660 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 22:52:40.545653 27660 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 22:52:40.545668 27660 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 22:52:40.545686 27660 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 22:52:40.547479 27660 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.147.212:33235
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790722213368738-14927-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790722213368738-14927-0/minicluster-data/master-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.14.147.212:46007/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.14.147.254:37205
--ranger_config_path=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790722213368738-14927-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790722213368738-14927-0/krb5kdc/kudu_127.14.147.254.keytab
--principal=kudu/127.14.147.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.147.254:37205
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790722213368738-14927-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.14.147.254
--webserver_port=38013
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27660
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790722213368738-14927-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 21:58:54 UTC on ccycloud.quasar-xguoti.root.comops.site
build id 4681390
I20260929 22:52:40.547734 27660 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 22:52:40.547955 27660 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 22:52:40.553054 27666 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 22:52:40.553056 27665 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 22:52:40.553213 27660 server_base.cc:1034] running on GCE node
W20260929 22:52:40.553056 27668 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 22:52:40.553805 27660 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 22:52:40.554203 27660 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 22:52:40.555388 27660 hybrid_clock.cc:648] HybridClock initialized: now 1790722360555375 us; error 30 us; skew 500 ppm
Sep 29 22:52:40 dist-test-slave-98mk krb5kdc[26697](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790722360, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.147.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 22:52:40.558524 27660 init.cc:377] Logged in from keytab as kudu/127.14.147.254@KRBTEST.COM (short username kudu)
I20260929 22:52:40.559103 27660 webserver.cc:460] Webserver started at http://127.14.147.254:38013/ using document root <none> and password file <none>
I20260929 22:52:40.559361 27660 fs_manager.cc:359] Metadata directory not provided
I20260929 22:52:40.559461 27660 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 29 22:52:40 dist-test-slave-98mk krb5kdc[26697](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790722360, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.147.254@KRBTEST.COM for HTTP/127.14.147.212@KRBTEST.COM
I20260929 22:52:40.617004 27660 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260929 22:52:40.617683 27676 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 22:52:40.617833 27660 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260929 22:52:40.617918 27660 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790722213368738-14927-0/minicluster-data/master-0/data,/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790722213368738-14927-0/minicluster-data/master-0/wal
uuid: "31ac56a028ae4c99bcfe0cde4b1efc91"
format_stamp: "Formatted at 2026-09-29 22:52:38 on dist-test-slave-98mk"
server_key: "3d409f837d75ae0f65292038860e6ebe"
server_key_iv: "0f03ebdb06c4f1f32301aded3561c7ef"
server_key_version: "kuduclusterkey@0"
I20260929 22:52:40.618322 27660 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790722213368738-14927-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790722213368738-14927-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790722213368738-14927-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 22:52:40.630450 27660 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 22:52:40.632371 27660 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 22:52:40.632491 27660 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 22:52:40.636930 27660 rpc_server.cc:307] RPC server started. Bound to: 127.14.147.254:37205
I20260929 22:52:40.636997 27728 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.147.254:37205 every 8 connection(s)
I20260929 22:52:40.637408 27660 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790722213368738-14927-0/minicluster-data/master-0/data/info.pb
I20260929 22:52:40.638684 27729 sys_catalog.cc:263] Verifying existing consensus state
I20260929 22:52:40.638692 14927 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhWOCLk/build/release/bin/kudu as pid 27660
I20260929 22:52:40.638952 14927 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhWOCLk/build/release/bin/kudu
/tmp/dist-test-taskhWOCLk/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790722213368738-14927-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790722213368738-14927-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790722213368738-14927-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790722213368738-14927-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.14.147.212:46007/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.147.193:44211
--local_ip_for_outbound_sockets=127.14.147.193
--tserver_master_addrs=127.14.147.254:37205
--webserver_port=41789
--webserver_interface=127.14.147.193
--keytab_file=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790722213368738-14927-0/krb5kdc/kudu_127.14.147.193.keytab
--principal=kudu/127.14.147.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.147.212:33235
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.14.147.212:46007/kms with env {KRB5CCNAME=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790722213368738-14927-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790722213368738-14927-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790722213368738-14927-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 22:52:40.639568 27729 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 31ac56a028ae4c99bcfe0cde4b1efc91: Bootstrap starting.
I20260929 22:52:40.649646 27729 log.cc:824] T 00000000000000000000000000000000 P 31ac56a028ae4c99bcfe0cde4b1efc91: Log is configured to *not* fsync() on all Append() calls
I20260929 22:52:40.652902 27729 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 31ac56a028ae4c99bcfe0cde4b1efc91: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260929 22:52:40.653220 27729 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 31ac56a028ae4c99bcfe0cde4b1efc91: Bootstrap complete.
I20260929 22:52:40.656468 27729 raft_consensus.cc:348] T 00000000000000000000000000000000 P 31ac56a028ae4c99bcfe0cde4b1efc91 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "31ac56a028ae4c99bcfe0cde4b1efc91" member_type: VOTER last_known_addr { host: "127.14.147.254" port: 37205 } }
I20260929 22:52:40.656805 27729 raft_consensus.cc:729] T 00000000000000000000000000000000 P 31ac56a028ae4c99bcfe0cde4b1efc91 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 31ac56a028ae4c99bcfe0cde4b1efc91, State: Initialized, Role: FOLLOWER
I20260929 22:52:40.656972 27729 consensus_queue.cc:260] T 00000000000000000000000000000000 P 31ac56a028ae4c99bcfe0cde4b1efc91 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "31ac56a028ae4c99bcfe0cde4b1efc91" member_type: VOTER last_known_addr { host: "127.14.147.254" port: 37205 } }
I20260929 22:52:40.657033 27729 raft_consensus.cc:388] T 00000000000000000000000000000000 P 31ac56a028ae4c99bcfe0cde4b1efc91 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 22:52:40.657076 27729 raft_consensus.cc:482] T 00000000000000000000000000000000 P 31ac56a028ae4c99bcfe0cde4b1efc91 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 22:52:40.657120 27729 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 31ac56a028ae4c99bcfe0cde4b1efc91 [term 1 FOLLOWER]: Advancing to term 2
I20260929 22:52:40.658537 27729 raft_consensus.cc:504] T 00000000000000000000000000000000 P 31ac56a028ae4c99bcfe0cde4b1efc91 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "31ac56a028ae4c99bcfe0cde4b1efc91" member_type: VOTER last_known_addr { host: "127.14.147.254" port: 37205 } }
I20260929 22:52:40.658651 27729 leader_election.cc:302] T 00000000000000000000000000000000 P 31ac56a028ae4c99bcfe0cde4b1efc91 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 31ac56a028ae4c99bcfe0cde4b1efc91; no voters:
I20260929 22:52:40.658825 27729 leader_election.cc:288] T 00000000000000000000000000000000 P 31ac56a028ae4c99bcfe0cde4b1efc91 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260929 22:52:40.658946 27733 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 31ac56a028ae4c99bcfe0cde4b1efc91 [term 2 FOLLOWER]: Leader election won for term 2
I20260929 22:52:40.659071 27729 sys_catalog.cc:564] T 00000000000000000000000000000000 P 31ac56a028ae4c99bcfe0cde4b1efc91 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 22:52:40.659108 27733 raft_consensus.cc:686] T 00000000000000000000000000000000 P 31ac56a028ae4c99bcfe0cde4b1efc91 [term 2 LEADER]: Becoming Leader. State: Replica: 31ac56a028ae4c99bcfe0cde4b1efc91, State: Running, Role: LEADER
I20260929 22:52:40.659217 27733 consensus_queue.cc:237] T 00000000000000000000000000000000 P 31ac56a028ae4c99bcfe0cde4b1efc91 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "31ac56a028ae4c99bcfe0cde4b1efc91" member_type: VOTER last_known_addr { host: "127.14.147.254" port: 37205 } }
I20260929 22:52:40.659505 27734 sys_catalog.cc:455] T 00000000000000000000000000000000 P 31ac56a028ae4c99bcfe0cde4b1efc91 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "31ac56a028ae4c99bcfe0cde4b1efc91" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "31ac56a028ae4c99bcfe0cde4b1efc91" member_type: VOTER last_known_addr { host: "127.14.147.254" port: 37205 } } }
I20260929 22:52:40.659579 27734 sys_catalog.cc:458] T 00000000000000000000000000000000 P 31ac56a028ae4c99bcfe0cde4b1efc91 [sys.catalog]: This master's current role is: LEADER
I20260929 22:52:40.659816 27735 sys_catalog.cc:455] T 00000000000000000000000000000000 P 31ac56a028ae4c99bcfe0cde4b1efc91 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 31ac56a028ae4c99bcfe0cde4b1efc91. Latest consensus state: current_term: 2 leader_uuid: "31ac56a028ae4c99bcfe0cde4b1efc91" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "31ac56a028ae4c99bcfe0cde4b1efc91" member_type: VOTER last_known_addr { host: "127.14.147.254" port: 37205 } } }
I20260929 22:52:40.659933 27735 sys_catalog.cc:458] T 00000000000000000000000000000000 P 31ac56a028ae4c99bcfe0cde4b1efc91 [sys.catalog]: This master's current role is: LEADER
I20260929 22:52:40.659964 27740 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 22:52:40.660440 27740 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 22:52:40.660557 27740 catalog_manager.cc:1259] Loaded cluster ID: fb7d2c791dad49a99bc01d9324f688be
I20260929 22:52:40.660593 27740 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 22:52:40.660913 27729 ranger_client.cc:318] Using overwritten properties file: /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790722213368738-14927-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260929 22:52:40.661589 27740 catalog_manager.cc:1504] Loading token signing keys...
I20260929 22:52:40.661767 27740 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 31ac56a028ae4c99bcfe0cde4b1efc91: Loaded TSK: 0
I20260929 22:52:40.661968 27740 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260929 22:52:40.754556 27731 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 22:52:40.754837 27731 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 22:52:40.754873 27731 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 22:52:40.756676 27731 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 22:52:40.756803 27731 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 22:52:40.756865 27731 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.147.193
I20260929 22:52:40.759178 27731 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.147.212:33235
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790722213368738-14927-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790722213368738-14927-0/minicluster-data/ts-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.14.147.212:46007/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790722213368738-14927-0/krb5kdc/kudu_127.14.147.193.keytab
--principal=kudu/127.14.147.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.147.193:44211
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790722213368738-14927-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.14.147.193
--webserver_port=41789
--tserver_master_addrs=127.14.147.254:37205
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27731
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.147.193
--log_dir=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790722213368738-14927-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 21:58:54 UTC on ccycloud.quasar-xguoti.root.comops.site
build id 4681390
I20260929 22:52:40.759670 27731 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 22:52:40.759954 27731 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 22:52:40.767190 27769 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 22:52:40.767567 27766 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 22:52:40.768111 27731 server_base.cc:1034] running on GCE node
W20260929 22:52:40.768739 27767 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 22:52:40.769089 27731 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 22:52:40.769425 27731 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 22:52:40.770594 27731 hybrid_clock.cc:648] HybridClock initialized: now 1790722360770559 us; error 49 us; skew 500 ppm
Sep 29 22:52:40 dist-test-slave-98mk krb5kdc[26697](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790722360, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.147.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 22:52:40.774334 27731 init.cc:377] Logged in from keytab as kudu/127.14.147.193@KRBTEST.COM (short username kudu)
I20260929 22:52:40.775094 27731 webserver.cc:460] Webserver started at http://127.14.147.193:41789/ using document root <none> and password file <none>
I20260929 22:52:40.775398 27731 fs_manager.cc:359] Metadata directory not provided
I20260929 22:52:40.775512 27731 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 29 22:52:40 dist-test-slave-98mk krb5kdc[26697](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790722360, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.147.193@KRBTEST.COM for HTTP/127.14.147.212@KRBTEST.COM
I20260929 22:52:40.806594 27731 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.001s sys 0.000s
I20260929 22:52:40.807808 27777 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 22:52:40.808020 27731 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260929 22:52:40.808101 27731 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790722213368738-14927-0/minicluster-data/ts-0/data,/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790722213368738-14927-0/minicluster-data/ts-0/wal
uuid: "738ca65869ba46809f58b052437e15e2"
format_stamp: "Formatted at 2026-09-29 22:52:39 on dist-test-slave-98mk"
server_key: "745a5cf4355e16629eb67479bdc86c7f"
server_key_iv: "cbf341961497f263dbb730b5d5dff745"
server_key_version: "kuduclusterkey@0"
I20260929 22:52:40.808570 27731 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790722213368738-14927-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790722213368738-14927-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790722213368738-14927-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 22:52:40.834169 27731 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 22:52:40.836645 27731 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 22:52:40.836823 27731 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 22:52:40.837044 27731 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 22:52:40.837448 27731 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 22:52:40.837499 27731 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 22:52:40.837533 27731 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 22:52:40.837558 27731 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 22:52:40.844692 27731 rpc_server.cc:307] RPC server started. Bound to: 127.14.147.193:44211
I20260929 22:52:40.845191 27731 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790722213368738-14927-0/minicluster-data/ts-0/data/info.pb
I20260929 22:52:40.848402 14927 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhWOCLk/build/release/bin/kudu as pid 27731
I20260929 22:52:40.848677 14927 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhWOCLk/build/release/bin/kudu
/tmp/dist-test-taskhWOCLk/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790722213368738-14927-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790722213368738-14927-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790722213368738-14927-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790722213368738-14927-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.14.147.212:46007/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.147.194:38247
--local_ip_for_outbound_sockets=127.14.147.194
--tserver_master_addrs=127.14.147.254:37205
--webserver_port=36753
--webserver_interface=127.14.147.194
--keytab_file=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790722213368738-14927-0/krb5kdc/kudu_127.14.147.194.keytab
--principal=kudu/127.14.147.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.147.212:33235
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.14.147.212:46007/kms with env {KRB5CCNAME=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790722213368738-14927-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790722213368738-14927-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790722213368738-14927-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 22:52:40.847108 27890 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.147.193:44211 every 8 connection(s)
Sep 29 22:52:40 dist-test-slave-98mk krb5kdc[26697](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790722360, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.147.193@KRBTEST.COM for kudu/127.14.147.254@KRBTEST.COM
I20260929 22:52:40.860018 27893 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:52:40.846324 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.147.254:37205 (local address 127.14.147.193:52443)
0929 22:52:40.847241 (+ 917us) negotiation.cc:107] Waiting for socket to connect
0929 22:52:40.847290 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0929 22:52:40.847923 (+ 633us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 22:52:40.848670 (+ 747us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 22:52:40.848675 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 22:52:40.848906 (+ 231us) client_negotiation.cc:189] Negotiated authn=SASL
0929 22:52:40.849189 (+ 283us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 22:52:40.849195 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 22:52:40.850949 (+ 1754us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 22:52:40.850950 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 22:52:40.852236 (+ 1286us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 22:52:40.852240 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 22:52:40.853226 (+ 986us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 22:52:40.853228 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 22:52:40.853369 (+ 141us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 22:52:40.853986 (+ 617us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 22:52:40.853997 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0929 22:52:40.855745 (+ 1748us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 22:52:40.857970 (+ 2225us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 22:52:40.857979 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 22:52:40.857982 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 22:52:40.858292 (+ 310us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 22:52:40.858612 (+ 320us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 22:52:40.858613 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 22:52:40.858613 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 22:52:40.858753 (+ 140us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 22:52:40.859270 (+ 517us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 22:52:40.859278 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 22:52:40.859474 (+ 196us) client_negotiation.cc:769] Sending connection context
0929 22:52:40.859577 (+ 103us) client_negotiation.cc:240] Negotiation successful
0929 22:52:40.859737 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":873,"thread_start_us":76,"threads_started":1}
I20260929 22:52:40.860893 27894 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:52:40.846670 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.147.193:52443 (local address 127.14.147.254:37205)
0929 22:52:40.848129 (+ 1459us) server_negotiation.cc:206] Beginning negotiation
0929 22:52:40.848138 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 22:52:40.848173 (+ 35us) server_negotiation.cc:407] Connection header received
0929 22:52:40.848316 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 22:52:40.848322 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 22:52:40.848379 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 22:52:40.848501 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0929 22:52:40.849318 (+ 817us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:52:40.850750 (+ 1432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:52:40.852343 (+ 1593us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:52:40.852991 (+ 648us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:52:40.853102 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 22:52:40.855892 (+ 2790us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 22:52:40.855897 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 22:52:40.855898 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 22:52:40.855911 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0929 22:52:40.857867 (+ 1956us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 22:52:40.858407 (+ 540us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 22:52:40.858411 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 22:52:40.858411 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 22:52:40.858531 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 22:52:40.858827 (+ 296us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 22:52:40.858828 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 22:52:40.858829 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 22:52:40.859160 (+ 331us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 22:52:40.859241 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0929 22:52:40.860551 (+ 1310us) server_negotiation.cc:299] Negotiation successful
0929 22:52:40.860691 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1405,"thread_start_us":69,"threads_started":1}
I20260929 22:52:40.862066 27891 heartbeater.cc:347] Connected to a master server at 127.14.147.254:37205
I20260929 22:52:40.862176 27891 heartbeater.cc:464] Registering TS with master...
I20260929 22:52:40.862382 27891 heartbeater.cc:511] Master 127.14.147.254:37205 requested a full tablet report, sending...
W20260929 22:52:40.863237 27891 heartbeater.cc:644] Failed to heartbeat to 127.14.147.254:37205 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260929 22:52:40.863329 27891 heartbeater.cc:464] Registering TS with master...
I20260929 22:52:40.863404 27891 heartbeater.cc:511] Master 127.14.147.254:37205 requested a full tablet report, sending...
I20260929 22:52:40.863762 27891 heartbeater.cc:464] Registering TS with master...
I20260929 22:52:40.863852 27891 heartbeater.cc:511] Master 127.14.147.254:37205 requested a full tablet report, sending...
W20260929 22:52:40.864145 27891 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260929 22:52:40.955196 27895 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 22:52:40.955456 27895 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 22:52:40.955499 27895 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 22:52:40.957239 27895 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 22:52:40.957309 27895 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 22:52:40.957351 27895 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.147.194
I20260929 22:52:40.959906 27895 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.147.212:33235
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790722213368738-14927-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790722213368738-14927-0/minicluster-data/ts-1/wal
--ranger_kms_url=invalid.host:1234/kms,127.14.147.212:46007/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790722213368738-14927-0/krb5kdc/kudu_127.14.147.194.keytab
--principal=kudu/127.14.147.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.147.194:38247
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790722213368738-14927-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.14.147.194
--webserver_port=36753
--tserver_master_addrs=127.14.147.254:37205
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27895
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.147.194
--log_dir=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790722213368738-14927-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 21:58:54 UTC on ccycloud.quasar-xguoti.root.comops.site
build id 4681390
I20260929 22:52:40.960166 27895 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 22:52:40.960407 27895 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 22:52:40.969738 27901 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 22:52:40.969738 27903 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 22:52:40.970479 27895 server_base.cc:1034] running on GCE node
W20260929 22:52:40.971570 27900 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 22:52:40.971858 27895 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 22:52:40.972150 27895 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 22:52:40.973335 27895 hybrid_clock.cc:648] HybridClock initialized: now 1790722360973322 us; error 32 us; skew 500 ppm
Sep 29 22:52:40 dist-test-slave-98mk krb5kdc[26697](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790722360, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.147.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 22:52:40.976576 27895 init.cc:377] Logged in from keytab as kudu/127.14.147.194@KRBTEST.COM (short username kudu)
I20260929 22:52:40.977044 27895 webserver.cc:460] Webserver started at http://127.14.147.194:36753/ using document root <none> and password file <none>
I20260929 22:52:40.977284 27895 fs_manager.cc:359] Metadata directory not provided
I20260929 22:52:40.977357 27895 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 29 22:52:40 dist-test-slave-98mk krb5kdc[26697](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790722360, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.147.194@KRBTEST.COM for HTTP/127.14.147.212@KRBTEST.COM
I20260929 22:52:41.000765 27895 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.001s sys 0.000s
I20260929 22:52:41.001775 27911 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 22:52:41.001926 27895 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260929 22:52:41.002010 27895 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790722213368738-14927-0/minicluster-data/ts-1/data,/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790722213368738-14927-0/minicluster-data/ts-1/wal
uuid: "fcf7eed1fd374116b677881d7eaadbd4"
format_stamp: "Formatted at 2026-09-29 22:52:40 on dist-test-slave-98mk"
server_key: "f88afea91e8cafe052ad8b6ee0130095"
server_key_iv: "41408663550f21bf2eead47819242edf"
server_key_version: "kuduclusterkey@0"
I20260929 22:52:41.002501 27895 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790722213368738-14927-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790722213368738-14927-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790722213368738-14927-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 22:52:41.020423 27895 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 22:52:41.022881 27895 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 22:52:41.023066 27895 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 22:52:41.023370 27895 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 22:52:41.023783 27895 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 22:52:41.023829 27895 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 22:52:41.023860 27895 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 22:52:41.023882 27895 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 22:52:41.031291 27895 rpc_server.cc:307] RPC server started. Bound to: 127.14.147.194:38247
I20260929 22:52:41.031523 28024 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.147.194:38247 every 8 connection(s)
I20260929 22:52:41.032097 27895 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790722213368738-14927-0/minicluster-data/ts-1/data/info.pb
I20260929 22:52:41.035542 14927 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhWOCLk/build/release/bin/kudu as pid 27895
I20260929 22:52:41.035984 14927 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhWOCLk/build/release/bin/kudu
/tmp/dist-test-taskhWOCLk/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790722213368738-14927-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790722213368738-14927-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790722213368738-14927-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790722213368738-14927-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.14.147.212:46007/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.147.195:41647
--local_ip_for_outbound_sockets=127.14.147.195
--tserver_master_addrs=127.14.147.254:37205
--webserver_port=36211
--webserver_interface=127.14.147.195
--keytab_file=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790722213368738-14927-0/krb5kdc/kudu_127.14.147.195.keytab
--principal=kudu/127.14.147.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.147.212:33235
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.14.147.212:46007/kms with env {KRB5CCNAME=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790722213368738-14927-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790722213368738-14927-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790722213368738-14927-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 29 22:52:41 dist-test-slave-98mk krb5kdc[26697](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790722360, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.147.194@KRBTEST.COM for kudu/127.14.147.254@KRBTEST.COM
I20260929 22:52:41.043601 27894 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:52:41.033202 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.147.194:56685 (local address 127.14.147.254:37205)
0929 22:52:41.033247 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0929 22:52:41.033249 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 22:52:41.033476 (+ 227us) server_negotiation.cc:407] Connection header received
0929 22:52:41.034141 (+ 665us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 22:52:41.034143 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 22:52:41.034158 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 22:52:41.034204 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0929 22:52:41.034768 (+ 564us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:52:41.035492 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:52:41.036896 (+ 1404us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:52:41.037205 (+ 309us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:52:41.037311 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 22:52:41.040127 (+ 2816us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 22:52:41.040128 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 22:52:41.040129 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 22:52:41.040138 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0929 22:52:41.041780 (+ 1642us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 22:52:41.042399 (+ 619us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 22:52:41.042401 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 22:52:41.042401 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 22:52:41.042464 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 22:52:41.042773 (+ 309us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 22:52:41.042774 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 22:52:41.042774 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 22:52:41.042935 (+ 161us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 22:52:41.042978 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0929 22:52:41.043362 (+ 384us) server_negotiation.cc:299] Negotiation successful
0929 22:52:41.043496 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260929 22:52:41.044169 28027 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:52:41.032920 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.147.254:37205 (local address 127.14.147.194:56685)
0929 22:52:41.033342 (+ 422us) negotiation.cc:107] Waiting for socket to connect
0929 22:52:41.033388 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0929 22:52:41.034026 (+ 638us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 22:52:41.034290 (+ 264us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 22:52:41.034294 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 22:52:41.034467 (+ 173us) client_negotiation.cc:189] Negotiated authn=SASL
0929 22:52:41.034660 (+ 193us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 22:52:41.034664 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 22:52:41.035599 (+ 935us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 22:52:41.035600 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 22:52:41.036793 (+ 1193us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 22:52:41.036797 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 22:52:41.037408 (+ 611us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 22:52:41.037409 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 22:52:41.037566 (+ 157us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 22:52:41.038161 (+ 595us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 22:52:41.038172 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0929 22:52:41.039983 (+ 1811us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 22:52:41.041902 (+ 1919us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 22:52:41.041911 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 22:52:41.041914 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 22:52:41.042306 (+ 392us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 22:52:41.042538 (+ 232us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 22:52:41.042539 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 22:52:41.042539 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 22:52:41.042695 (+ 156us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 22:52:41.043025 (+ 330us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 22:52:41.043032 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 22:52:41.043209 (+ 177us) client_negotiation.cc:769] Sending connection context
0929 22:52:41.043748 (+ 539us) client_negotiation.cc:240] Negotiation successful
0929 22:52:41.043933 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":371,"thread_start_us":80,"threads_started":1}
I20260929 22:52:41.044693 28025 heartbeater.cc:347] Connected to a master server at 127.14.147.254:37205
I20260929 22:52:41.044814 28025 heartbeater.cc:464] Registering TS with master...
I20260929 22:52:41.045033 28025 heartbeater.cc:511] Master 127.14.147.254:37205 requested a full tablet report, sending...
W20260929 22:52:41.045883 28025 heartbeater.cc:644] Failed to heartbeat to 127.14.147.254:37205 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260929 22:52:41.045956 28025 heartbeater.cc:464] Registering TS with master...
I20260929 22:52:41.046026 28025 heartbeater.cc:511] Master 127.14.147.254:37205 requested a full tablet report, sending...
I20260929 22:52:41.046427 28025 heartbeater.cc:464] Registering TS with master...
I20260929 22:52:41.046499 28025 heartbeater.cc:511] Master 127.14.147.254:37205 requested a full tablet report, sending...
W20260929 22:52:41.046828 28025 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260929 22:52:41.148281 28028 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 22:52:41.148571 28028 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 22:52:41.148608 28028 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 22:52:41.150475 28028 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 22:52:41.150530 28028 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 22:52:41.150574 28028 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.147.195
I20260929 22:52:41.153036 28028 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.147.212:33235
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790722213368738-14927-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790722213368738-14927-0/minicluster-data/ts-2/wal
--ranger_kms_url=invalid.host:1234/kms,127.14.147.212:46007/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790722213368738-14927-0/krb5kdc/kudu_127.14.147.195.keytab
--principal=kudu/127.14.147.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.147.195:41647
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790722213368738-14927-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.14.147.195
--webserver_port=36211
--tserver_master_addrs=127.14.147.254:37205
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28028
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.147.195
--log_dir=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790722213368738-14927-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 21:58:54 UTC on ccycloud.quasar-xguoti.root.comops.site
build id 4681390
I20260929 22:52:41.153326 28028 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 22:52:41.153585 28028 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 22:52:41.160477 28035 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 22:52:41.160660 28034 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 22:52:41.160555 28028 server_base.cc:1034] running on GCE node
W20260929 22:52:41.160497 28037 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 22:52:41.160954 28028 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 22:52:41.161194 28028 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 22:52:41.162397 28028 hybrid_clock.cc:648] HybridClock initialized: now 1790722361162384 us; error 27 us; skew 500 ppm
Sep 29 22:52:41 dist-test-slave-98mk krb5kdc[26697](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790722361, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.147.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 22:52:41.164990 28028 init.cc:377] Logged in from keytab as kudu/127.14.147.195@KRBTEST.COM (short username kudu)
I20260929 22:52:41.165508 28028 webserver.cc:460] Webserver started at http://127.14.147.195:36211/ using document root <none> and password file <none>
I20260929 22:52:41.165773 28028 fs_manager.cc:359] Metadata directory not provided
I20260929 22:52:41.165840 28028 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 29 22:52:41 dist-test-slave-98mk krb5kdc[26697](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790722361, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.147.195@KRBTEST.COM for HTTP/127.14.147.212@KRBTEST.COM
I20260929 22:52:41.191593 28028 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.001s sys 0.000s
I20260929 22:52:41.193578 28045 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 22:52:41.193796 28028 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260929 22:52:41.193902 28028 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790722213368738-14927-0/minicluster-data/ts-2/data,/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790722213368738-14927-0/minicluster-data/ts-2/wal
uuid: "a143941d3e8349a2941b79eb89915ba4"
format_stamp: "Formatted at 2026-09-29 22:52:40 on dist-test-slave-98mk"
server_key: "409f10ac47621fa46fd3f3f7044c847a"
server_key_iv: "20e9bcb5f455c8bf4f89d84978bf6fad"
server_key_version: "kuduclusterkey@0"
I20260929 22:52:41.194265 28028 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790722213368738-14927-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790722213368738-14927-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790722213368738-14927-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 22:52:41.208424 28028 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 22:52:41.210500 28028 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 22:52:41.210655 28028 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 22:52:41.210904 28028 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 22:52:41.211236 28028 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 22:52:41.211272 28028 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 22:52:41.211302 28028 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 22:52:41.211323 28028 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 22:52:41.218281 28028 rpc_server.cc:307] RPC server started. Bound to: 127.14.147.195:41647
I20260929 22:52:41.218376 28158 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.147.195:41647 every 8 connection(s)
I20260929 22:52:41.218765 28028 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790722213368738-14927-0/minicluster-data/ts-2/data/info.pb
I20260929 22:52:41.222327 14927 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhWOCLk/build/release/bin/kudu as pid 28028
Sep 29 22:52:41 dist-test-slave-98mk krb5kdc[26697](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790722361, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.147.195@KRBTEST.COM for kudu/127.14.147.254@KRBTEST.COM
I20260929 22:52:41.232502 28162 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:52:41.222762 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41660 (local address 127.14.147.254:37205)
0929 22:52:41.222872 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0929 22:52:41.222874 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 22:52:41.223202 (+ 328us) server_negotiation.cc:407] Connection header received
0929 22:52:41.223239 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 22:52:41.223240 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 22:52:41.223251 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 22:52:41.223305 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0929 22:52:41.224033 (+ 728us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:52:41.224591 (+ 558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:52:41.225656 (+ 1065us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:52:41.225879 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:52:41.225934 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 22:52:41.229179 (+ 3245us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 22:52:41.229181 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 22:52:41.229181 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 22:52:41.229199 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0929 22:52:41.230631 (+ 1432us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 22:52:41.230972 (+ 341us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 22:52:41.230973 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 22:52:41.230974 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 22:52:41.231035 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 22:52:41.231927 (+ 892us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 22:52:41.231928 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 22:52:41.231928 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 22:52:41.232053 (+ 125us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 22:52:41.232241 (+ 188us) server_negotiation.cc:1035] Waiting for connection context
0929 22:52:41.232287 (+ 46us) server_negotiation.cc:299] Negotiation successful
0929 22:52:41.232394 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":89,"thread_start_us":58,"threads_started":1}
I20260929 22:52:41.234509 28161 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:52:41.219529 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.147.254:37205 (local address 127.14.147.195:33101)
0929 22:52:41.219689 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0929 22:52:41.219747 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0929 22:52:41.220229 (+ 482us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 22:52:41.220465 (+ 236us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 22:52:41.220469 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 22:52:41.220656 (+ 187us) client_negotiation.cc:189] Negotiated authn=SASL
0929 22:52:41.220861 (+ 205us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 22:52:41.220865 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 22:52:41.221884 (+ 1019us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 22:52:41.221885 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 22:52:41.223006 (+ 1121us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 22:52:41.223010 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 22:52:41.223597 (+ 587us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 22:52:41.223599 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 22:52:41.223765 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 22:52:41.224330 (+ 565us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 22:52:41.224342 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0929 22:52:41.229061 (+ 4719us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 22:52:41.232625 (+ 3564us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 22:52:41.232637 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 22:52:41.232640 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 22:52:41.232985 (+ 345us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 22:52:41.233231 (+ 246us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 22:52:41.233232 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 22:52:41.233232 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 22:52:41.233390 (+ 158us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 22:52:41.233790 (+ 400us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 22:52:41.233796 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 22:52:41.233977 (+ 181us) client_negotiation.cc:769] Sending connection context
0929 22:52:41.234087 (+ 110us) client_negotiation.cc:240] Negotiation successful
0929 22:52:41.234270 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":72,"threads_started":1}
I20260929 22:52:41.234997 27894 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:52:41.219819 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.147.195:33101 (local address 127.14.147.254:37205)
0929 22:52:41.219866 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0929 22:52:41.219868 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 22:52:41.219878 (+ 10us) server_negotiation.cc:407] Connection header received
0929 22:52:41.220332 (+ 454us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 22:52:41.220333 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 22:52:41.220346 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 22:52:41.220395 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0929 22:52:41.220982 (+ 587us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:52:41.221761 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:52:41.223099 (+ 1338us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:52:41.223392 (+ 293us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:52:41.223507 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 22:52:41.229184 (+ 5677us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 22:52:41.229186 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 22:52:41.229186 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 22:52:41.229194 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0929 22:52:41.232456 (+ 3262us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 22:52:41.233072 (+ 616us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 22:52:41.233074 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 22:52:41.233074 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 22:52:41.233133 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 22:52:41.233562 (+ 429us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 22:52:41.233564 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 22:52:41.233564 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 22:52:41.233697 (+ 133us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 22:52:41.233761 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0929 22:52:41.234820 (+ 1059us) server_negotiation.cc:299] Negotiation successful
0929 22:52:41.234911 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260929 22:52:41.235337 28159 heartbeater.cc:347] Connected to a master server at 127.14.147.254:37205
I20260929 22:52:41.235468 28159 heartbeater.cc:464] Registering TS with master...
I20260929 22:52:41.235668 28159 heartbeater.cc:511] Master 127.14.147.254:37205 requested a full tablet report, sending...
W20260929 22:52:41.236323 28159 heartbeater.cc:644] Failed to heartbeat to 127.14.147.254:37205 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260929 22:52:41.236397 28159 heartbeater.cc:464] Registering TS with master...
I20260929 22:52:41.236454 28159 heartbeater.cc:511] Master 127.14.147.254:37205 requested a full tablet report, sending...
I20260929 22:52:41.236794 28159 heartbeater.cc:464] Registering TS with master...
I20260929 22:52:41.236873 28159 heartbeater.cc:511] Master 127.14.147.254:37205 requested a full tablet report, sending...
W20260929 22:52:41.237200 28159 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
Sep 29 22:52:41 dist-test-slave-98mk krb5kdc[26697](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790722361, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.147.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 22:52:41.864341 27891 heartbeater.cc:464] Registering TS with master...
I20260929 22:52:41.864514 27891 heartbeater.cc:511] Master 127.14.147.254:37205 requested a full tablet report, sending...
I20260929 22:52:41.865011 27693 ts_manager.cc:194] Registered new tserver with Master: 738ca65869ba46809f58b052437e15e2 (127.14.147.193:44211)
I20260929 22:52:41.865999 27693 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.147.193@KRBTEST.COM'} at 127.14.147.193:52443
I20260929 22:52:42.047037 28025 heartbeater.cc:464] Registering TS with master...
I20260929 22:52:42.047214 28025 heartbeater.cc:511] Master 127.14.147.254:37205 requested a full tablet report, sending...
I20260929 22:52:42.047572 27693 ts_manager.cc:194] Registered new tserver with Master: fcf7eed1fd374116b677881d7eaadbd4 (127.14.147.194:38247)
I20260929 22:52:42.048130 27693 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.147.194@KRBTEST.COM'} at 127.14.147.194:56685
Sep 29 22:52:42 dist-test-slave-98mk krb5kdc[26697](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790722361, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.147.254@KRBTEST.COM for HTTP/127.14.147.212@KRBTEST.COM
I20260929 22:52:42.237389 28159 heartbeater.cc:464] Registering TS with master...
I20260929 22:52:42.237561 28159 heartbeater.cc:511] Master 127.14.147.254:37205 requested a full tablet report, sending...
I20260929 22:52:42.237937 27693 ts_manager.cc:194] Registered new tserver with Master: a143941d3e8349a2941b79eb89915ba4 (127.14.147.195:41647)
I20260929 22:52:42.238431 27693 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.147.195@KRBTEST.COM'} at 127.14.147.195:33101
I20260929 22:52:42.239164 14927 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 22:52:42.248654 28181 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:52:42.240189 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41664 (local address 127.14.147.254:37205)
0929 22:52:42.240301 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0929 22:52:42.240303 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 22:52:42.240408 (+ 105us) server_negotiation.cc:407] Connection header received
0929 22:52:42.240461 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 22:52:42.240462 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 22:52:42.240474 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 22:52:42.240537 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0929 22:52:42.241169 (+ 632us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:52:42.241916 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:52:42.242962 (+ 1046us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:52:42.243266 (+ 304us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:52:42.243324 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 22:52:42.244841 (+ 1517us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 22:52:42.244843 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 22:52:42.244843 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 22:52:42.244851 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0929 22:52:42.246824 (+ 1973us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 22:52:42.247359 (+ 535us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 22:52:42.247361 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 22:52:42.247361 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 22:52:42.247515 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 22:52:42.247854 (+ 339us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 22:52:42.247857 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 22:52:42.247857 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 22:52:42.248050 (+ 193us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 22:52:42.248179 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0929 22:52:42.248431 (+ 252us) server_negotiation.cc:299] Negotiation successful
0929 22:52:42.248543 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":87,"thread_start_us":59,"threads_started":1}
I20260929 22:52:42.251344 27693 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41664:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 22:52:42.253818 27693 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 22:52:42.263994 28190 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:52:42.256904 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.147.195:41647 (local address 127.0.0.1:56004)
0929 22:52:42.257282 (+ 378us) negotiation.cc:107] Waiting for socket to connect
0929 22:52:42.257329 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0929 22:52:42.257399 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 22:52:42.259816 (+ 2417us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 22:52:42.259819 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 22:52:42.259822 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 22:52:42.259910 (+ 88us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 22:52:42.259912 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 22:52:42.261407 (+ 1495us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 22:52:42.261408 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 22:52:42.262636 (+ 1228us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 22:52:42.262638 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 22:52:42.263686 (+ 1048us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 22:52:42.263687 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 22:52:42.263772 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 22:52:42.263785 (+ 13us) client_negotiation.cc:769] Sending connection context
0929 22:52:42.263847 (+ 62us) client_negotiation.cc:240] Negotiation successful
0929 22:52:42.263903 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":348,"thread_start_us":76,"threads_started":1}
I20260929 22:52:42.264627 28191 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:52:42.257156 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56004 (local address 127.14.147.195:41647)
0929 22:52:42.259551 (+ 2395us) server_negotiation.cc:206] Beginning negotiation
0929 22:52:42.259557 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 22:52:42.259569 (+ 12us) server_negotiation.cc:407] Connection header received
0929 22:52:42.259606 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 22:52:42.259608 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 22:52:42.259625 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 22:52:42.259973 (+ 348us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 22:52:42.260059 (+ 86us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:52:42.261331 (+ 1272us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:52:42.262732 (+ 1401us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:52:42.263526 (+ 794us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:52:42.263592 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 22:52:42.263653 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0929 22:52:42.264497 (+ 844us) server_negotiation.cc:299] Negotiation successful
0929 22:52:42.264550 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2355,"thread_start_us":54,"threads_started":1}
I20260929 22:52:42.265487 28093 tablet_service.cc:1467] Processing CreateTablet for tablet 1188e6d8f5e149a3995fa1b033ecb1c9 (DEFAULT_TABLE table=test-table [id=d88954e9def14856ab6512e283f80154]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 22:52:42.265904 28188 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:52:42.257137 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.147.194:38247 (local address 127.0.0.1:53844)
0929 22:52:42.260831 (+ 3694us) negotiation.cc:107] Waiting for socket to connect
0929 22:52:42.260842 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0929 22:52:42.260904 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 22:52:42.261161 (+ 257us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 22:52:42.261163 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 22:52:42.261165 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 22:52:42.261212 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 22:52:42.261214 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 22:52:42.263007 (+ 1793us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 22:52:42.263009 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 22:52:42.264228 (+ 1219us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 22:52:42.264232 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 22:52:42.265607 (+ 1375us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 22:52:42.265608 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 22:52:42.265735 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 22:52:42.265740 (+ 5us) client_negotiation.cc:769] Sending connection context
0929 22:52:42.265790 (+ 50us) client_negotiation.cc:240] Negotiation successful
0929 22:52:42.265836 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3671,"thread_start_us":76,"threads_started":1}
I20260929 22:52:42.265915 28093 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1188e6d8f5e149a3995fa1b033ecb1c9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 22:52:42.266255 28192 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:52:42.257341 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53844 (local address 127.14.147.194:38247)
0929 22:52:42.257870 (+ 529us) server_negotiation.cc:206] Beginning negotiation
0929 22:52:42.257875 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 22:52:42.260917 (+ 3042us) server_negotiation.cc:407] Connection header received
0929 22:52:42.260962 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 22:52:42.260964 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 22:52:42.260982 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 22:52:42.261066 (+ 84us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 22:52:42.261339 (+ 273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:52:42.262898 (+ 1559us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:52:42.264350 (+ 1452us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:52:42.265511 (+ 1161us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:52:42.265593 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 22:52:42.265649 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0929 22:52:42.266136 (+ 487us) server_negotiation.cc:299] Negotiation successful
0929 22:52:42.266184 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":493,"thread_start_us":76,"threads_started":1}
I20260929 22:52:42.269196 28193 tablet_bootstrap.cc:492] T 1188e6d8f5e149a3995fa1b033ecb1c9 P a143941d3e8349a2941b79eb89915ba4: Bootstrap starting.
I20260929 22:52:42.269596 28187 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:52:42.256940 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.147.193:44211 (local address 127.0.0.1:60922)
0929 22:52:42.257924 (+ 984us) negotiation.cc:107] Waiting for socket to connect
0929 22:52:42.257933 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0929 22:52:42.257982 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 22:52:42.265687 (+ 7705us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 22:52:42.265689 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 22:52:42.265692 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 22:52:42.265750 (+ 58us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 22:52:42.265751 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 22:52:42.267082 (+ 1331us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 22:52:42.267084 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 22:52:42.268269 (+ 1185us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 22:52:42.268272 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 22:52:42.269254 (+ 982us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 22:52:42.269256 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 22:52:42.269423 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 22:52:42.269427 (+ 4us) client_negotiation.cc:769] Sending connection context
0929 22:52:42.269477 (+ 50us) client_negotiation.cc:240] Negotiation successful
0929 22:52:42.269520 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":968,"thread_start_us":76,"threads_started":1}
I20260929 22:52:42.269711 27941 tablet_service.cc:1467] Processing CreateTablet for tablet 1188e6d8f5e149a3995fa1b033ecb1c9 (DEFAULT_TABLE table=test-table [id=d88954e9def14856ab6512e283f80154]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 22:52:42.269871 28193 tablet_bootstrap.cc:654] T 1188e6d8f5e149a3995fa1b033ecb1c9 P a143941d3e8349a2941b79eb89915ba4: Neither blocks nor log segments found. Creating new log.
I20260929 22:52:42.270007 28189 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:52:42.257038 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60922 (local address 127.14.147.193:44211)
0929 22:52:42.264916 (+ 7878us) server_negotiation.cc:206] Beginning negotiation
0929 22:52:42.264922 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 22:52:42.264933 (+ 11us) server_negotiation.cc:407] Connection header received
0929 22:52:42.264970 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 22:52:42.264972 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 22:52:42.264989 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 22:52:42.265067 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 22:52:42.265837 (+ 770us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:52:42.266998 (+ 1161us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:52:42.268383 (+ 1385us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:52:42.269091 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:52:42.269164 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 22:52:42.269217 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0929 22:52:42.269830 (+ 613us) server_negotiation.cc:299] Negotiation successful
0929 22:52:42.269879 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7837,"thread_start_us":52,"threads_started":1}
I20260929 22:52:42.270085 27941 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1188e6d8f5e149a3995fa1b033ecb1c9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 22:52:42.270164 28193 log.cc:824] T 1188e6d8f5e149a3995fa1b033ecb1c9 P a143941d3e8349a2941b79eb89915ba4: Log is configured to *not* fsync() on all Append() calls
I20260929 22:52:42.270874 27825 tablet_service.cc:1467] Processing CreateTablet for tablet 1188e6d8f5e149a3995fa1b033ecb1c9 (DEFAULT_TABLE table=test-table [id=d88954e9def14856ab6512e283f80154]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 22:52:42.271153 27825 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1188e6d8f5e149a3995fa1b033ecb1c9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 22:52:42.273965 28195 tablet_bootstrap.cc:492] T 1188e6d8f5e149a3995fa1b033ecb1c9 P fcf7eed1fd374116b677881d7eaadbd4: Bootstrap starting.
I20260929 22:52:42.274621 28195 tablet_bootstrap.cc:654] T 1188e6d8f5e149a3995fa1b033ecb1c9 P fcf7eed1fd374116b677881d7eaadbd4: Neither blocks nor log segments found. Creating new log.
I20260929 22:52:42.274955 28195 log.cc:824] T 1188e6d8f5e149a3995fa1b033ecb1c9 P fcf7eed1fd374116b677881d7eaadbd4: Log is configured to *not* fsync() on all Append() calls
I20260929 22:52:42.275154 28193 tablet_bootstrap.cc:492] T 1188e6d8f5e149a3995fa1b033ecb1c9 P a143941d3e8349a2941b79eb89915ba4: No bootstrap required, opened a new log
I20260929 22:52:42.275236 28193 ts_tablet_manager.cc:1397] T 1188e6d8f5e149a3995fa1b033ecb1c9 P a143941d3e8349a2941b79eb89915ba4: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.000s
I20260929 22:52:42.275296 28196 tablet_bootstrap.cc:492] T 1188e6d8f5e149a3995fa1b033ecb1c9 P 738ca65869ba46809f58b052437e15e2: Bootstrap starting.
I20260929 22:52:42.275943 28196 tablet_bootstrap.cc:654] T 1188e6d8f5e149a3995fa1b033ecb1c9 P 738ca65869ba46809f58b052437e15e2: Neither blocks nor log segments found. Creating new log.
I20260929 22:52:42.276239 28196 log.cc:824] T 1188e6d8f5e149a3995fa1b033ecb1c9 P 738ca65869ba46809f58b052437e15e2: Log is configured to *not* fsync() on all Append() calls
I20260929 22:52:42.277045 28195 tablet_bootstrap.cc:492] T 1188e6d8f5e149a3995fa1b033ecb1c9 P fcf7eed1fd374116b677881d7eaadbd4: No bootstrap required, opened a new log
I20260929 22:52:42.277125 28195 ts_tablet_manager.cc:1397] T 1188e6d8f5e149a3995fa1b033ecb1c9 P fcf7eed1fd374116b677881d7eaadbd4: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260929 22:52:42.277432 28193 raft_consensus.cc:348] T 1188e6d8f5e149a3995fa1b033ecb1c9 P a143941d3e8349a2941b79eb89915ba4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a143941d3e8349a2941b79eb89915ba4" member_type: VOTER last_known_addr { host: "127.14.147.195" port: 41647 } } peers { permanent_uuid: "738ca65869ba46809f58b052437e15e2" member_type: VOTER last_known_addr { host: "127.14.147.193" port: 44211 } } peers { permanent_uuid: "fcf7eed1fd374116b677881d7eaadbd4" member_type: VOTER last_known_addr { host: "127.14.147.194" port: 38247 } }
I20260929 22:52:42.277609 28193 raft_consensus.cc:374] T 1188e6d8f5e149a3995fa1b033ecb1c9 P a143941d3e8349a2941b79eb89915ba4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 22:52:42.277648 28193 raft_consensus.cc:729] T 1188e6d8f5e149a3995fa1b033ecb1c9 P a143941d3e8349a2941b79eb89915ba4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a143941d3e8349a2941b79eb89915ba4, State: Initialized, Role: FOLLOWER
I20260929 22:52:42.277784 28193 consensus_queue.cc:260] T 1188e6d8f5e149a3995fa1b033ecb1c9 P a143941d3e8349a2941b79eb89915ba4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a143941d3e8349a2941b79eb89915ba4" member_type: VOTER last_known_addr { host: "127.14.147.195" port: 41647 } } peers { permanent_uuid: "738ca65869ba46809f58b052437e15e2" member_type: VOTER last_known_addr { host: "127.14.147.193" port: 44211 } } peers { permanent_uuid: "fcf7eed1fd374116b677881d7eaadbd4" member_type: VOTER last_known_addr { host: "127.14.147.194" port: 38247 } }
I20260929 22:52:42.278055 28193 ts_tablet_manager.cc:1428] T 1188e6d8f5e149a3995fa1b033ecb1c9 P a143941d3e8349a2941b79eb89915ba4: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260929 22:52:42.278173 28159 heartbeater.cc:503] Master 127.14.147.254:37205 was elected leader, sending a full tablet report...
I20260929 22:52:42.279381 28195 raft_consensus.cc:348] T 1188e6d8f5e149a3995fa1b033ecb1c9 P fcf7eed1fd374116b677881d7eaadbd4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a143941d3e8349a2941b79eb89915ba4" member_type: VOTER last_known_addr { host: "127.14.147.195" port: 41647 } } peers { permanent_uuid: "738ca65869ba46809f58b052437e15e2" member_type: VOTER last_known_addr { host: "127.14.147.193" port: 44211 } } peers { permanent_uuid: "fcf7eed1fd374116b677881d7eaadbd4" member_type: VOTER last_known_addr { host: "127.14.147.194" port: 38247 } }
I20260929 22:52:42.279589 28195 raft_consensus.cc:374] T 1188e6d8f5e149a3995fa1b033ecb1c9 P fcf7eed1fd374116b677881d7eaadbd4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 22:52:42.279631 28195 raft_consensus.cc:729] T 1188e6d8f5e149a3995fa1b033ecb1c9 P fcf7eed1fd374116b677881d7eaadbd4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fcf7eed1fd374116b677881d7eaadbd4, State: Initialized, Role: FOLLOWER
I20260929 22:52:42.279765 28195 consensus_queue.cc:260] T 1188e6d8f5e149a3995fa1b033ecb1c9 P fcf7eed1fd374116b677881d7eaadbd4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a143941d3e8349a2941b79eb89915ba4" member_type: VOTER last_known_addr { host: "127.14.147.195" port: 41647 } } peers { permanent_uuid: "738ca65869ba46809f58b052437e15e2" member_type: VOTER last_known_addr { host: "127.14.147.193" port: 44211 } } peers { permanent_uuid: "fcf7eed1fd374116b677881d7eaadbd4" member_type: VOTER last_known_addr { host: "127.14.147.194" port: 38247 } }
I20260929 22:52:42.280048 28195 ts_tablet_manager.cc:1428] T 1188e6d8f5e149a3995fa1b033ecb1c9 P fcf7eed1fd374116b677881d7eaadbd4: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260929 22:52:42.280170 28025 heartbeater.cc:503] Master 127.14.147.254:37205 was elected leader, sending a full tablet report...
I20260929 22:52:42.280177 28196 tablet_bootstrap.cc:492] T 1188e6d8f5e149a3995fa1b033ecb1c9 P 738ca65869ba46809f58b052437e15e2: No bootstrap required, opened a new log
I20260929 22:52:42.280243 28196 ts_tablet_manager.cc:1397] T 1188e6d8f5e149a3995fa1b033ecb1c9 P 738ca65869ba46809f58b052437e15e2: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.000s
I20260929 22:52:42.282749 28196 raft_consensus.cc:348] T 1188e6d8f5e149a3995fa1b033ecb1c9 P 738ca65869ba46809f58b052437e15e2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a143941d3e8349a2941b79eb89915ba4" member_type: VOTER last_known_addr { host: "127.14.147.195" port: 41647 } } peers { permanent_uuid: "738ca65869ba46809f58b052437e15e2" member_type: VOTER last_known_addr { host: "127.14.147.193" port: 44211 } } peers { permanent_uuid: "fcf7eed1fd374116b677881d7eaadbd4" member_type: VOTER last_known_addr { host: "127.14.147.194" port: 38247 } }
I20260929 22:52:42.282940 28196 raft_consensus.cc:374] T 1188e6d8f5e149a3995fa1b033ecb1c9 P 738ca65869ba46809f58b052437e15e2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 22:52:42.282974 28196 raft_consensus.cc:729] T 1188e6d8f5e149a3995fa1b033ecb1c9 P 738ca65869ba46809f58b052437e15e2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 738ca65869ba46809f58b052437e15e2, State: Initialized, Role: FOLLOWER
I20260929 22:52:42.283083 28196 consensus_queue.cc:260] T 1188e6d8f5e149a3995fa1b033ecb1c9 P 738ca65869ba46809f58b052437e15e2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a143941d3e8349a2941b79eb89915ba4" member_type: VOTER last_known_addr { host: "127.14.147.195" port: 41647 } } peers { permanent_uuid: "738ca65869ba46809f58b052437e15e2" member_type: VOTER last_known_addr { host: "127.14.147.193" port: 44211 } } peers { permanent_uuid: "fcf7eed1fd374116b677881d7eaadbd4" member_type: VOTER last_known_addr { host: "127.14.147.194" port: 38247 } }
I20260929 22:52:42.283352 28196 ts_tablet_manager.cc:1428] T 1188e6d8f5e149a3995fa1b033ecb1c9 P 738ca65869ba46809f58b052437e15e2: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260929 22:52:42.283396 27891 heartbeater.cc:503] Master 127.14.147.254:37205 was elected leader, sending a full tablet report...
W20260929 22:52:42.284153 28026 tablet.cc:2369] T 1188e6d8f5e149a3995fa1b033ecb1c9 P fcf7eed1fd374116b677881d7eaadbd4: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260929 22:52:42.348098 27892 tablet.cc:2369] T 1188e6d8f5e149a3995fa1b033ecb1c9 P 738ca65869ba46809f58b052437e15e2: Can't schedule compaction. Clean time has not been advanced past its initial value.
Sep 29 22:52:42 dist-test-slave-98mk krb5kdc[26697](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790722361, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.147.254@KRBTEST.COM for HTTP/127.14.147.212@KRBTEST.COM
W20260929 22:52:42.469982 28160 tablet.cc:2369] T 1188e6d8f5e149a3995fa1b033ecb1c9 P a143941d3e8349a2941b79eb89915ba4: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 22:52:42.610893 28199 raft_consensus.cc:482] T 1188e6d8f5e149a3995fa1b033ecb1c9 P a143941d3e8349a2941b79eb89915ba4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 22:52:42.611047 28199 raft_consensus.cc:504] T 1188e6d8f5e149a3995fa1b033ecb1c9 P a143941d3e8349a2941b79eb89915ba4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a143941d3e8349a2941b79eb89915ba4" member_type: VOTER last_known_addr { host: "127.14.147.195" port: 41647 } } peers { permanent_uuid: "738ca65869ba46809f58b052437e15e2" member_type: VOTER last_known_addr { host: "127.14.147.193" port: 44211 } } peers { permanent_uuid: "fcf7eed1fd374116b677881d7eaadbd4" member_type: VOTER last_known_addr { host: "127.14.147.194" port: 38247 } }
I20260929 22:52:42.611634 28199 leader_election.cc:288] T 1188e6d8f5e149a3995fa1b033ecb1c9 P a143941d3e8349a2941b79eb89915ba4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 738ca65869ba46809f58b052437e15e2 (127.14.147.193:44211), fcf7eed1fd374116b677881d7eaadbd4 (127.14.147.194:38247)
I20260929 22:52:42.616158 28189 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:52:42.612448 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.147.195:33059 (local address 127.14.147.193:44211)
0929 22:52:42.612513 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0929 22:52:42.612515 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 22:52:42.612556 (+ 41us) server_negotiation.cc:407] Connection header received
0929 22:52:42.612644 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 22:52:42.612644 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 22:52:42.612653 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 22:52:42.612682 (+ 29us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 22:52:42.613036 (+ 354us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:52:42.613755 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:52:42.615182 (+ 1427us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:52:42.615762 (+ 580us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:52:42.615821 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 22:52:42.615833 (+ 12us) server_negotiation.cc:1035] Waiting for connection context
0929 22:52:42.616040 (+ 207us) server_negotiation.cc:299] Negotiation successful
0929 22:52:42.616083 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260929 22:52:42.616420 28204 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:52:42.612287 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.147.193:44211 (local address 127.14.147.195:33059)
0929 22:52:42.612507 (+ 220us) negotiation.cc:107] Waiting for socket to connect
0929 22:52:42.612515 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0929 22:52:42.612573 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 22:52:42.612750 (+ 177us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 22:52:42.612751 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 22:52:42.612753 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 22:52:42.612965 (+ 212us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 22:52:42.612966 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 22:52:42.613884 (+ 918us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 22:52:42.613885 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 22:52:42.615084 (+ 1199us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 22:52:42.615087 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 22:52:42.615901 (+ 814us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 22:52:42.615902 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 22:52:42.615964 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 22:52:42.615967 (+ 3us) client_negotiation.cc:769] Sending connection context
0929 22:52:42.616277 (+ 310us) client_negotiation.cc:240] Negotiation successful
0929 22:52:42.616322 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":207,"mutex_wait_us":147,"thread_start_us":75,"threads_started":1}
I20260929 22:52:42.616719 27831 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1188e6d8f5e149a3995fa1b033ecb1c9" candidate_uuid: "a143941d3e8349a2941b79eb89915ba4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "738ca65869ba46809f58b052437e15e2" is_pre_election: true
I20260929 22:52:42.616904 27831 raft_consensus.cc:2454] T 1188e6d8f5e149a3995fa1b033ecb1c9 P 738ca65869ba46809f58b052437e15e2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a143941d3e8349a2941b79eb89915ba4 in term 0.
I20260929 22:52:42.617129 28049 leader_election.cc:302] T 1188e6d8f5e149a3995fa1b033ecb1c9 P a143941d3e8349a2941b79eb89915ba4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 738ca65869ba46809f58b052437e15e2, a143941d3e8349a2941b79eb89915ba4; no voters:
I20260929 22:52:42.617197 28192 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:52:42.612157 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.147.195:56983 (local address 127.14.147.194:38247)
0929 22:52:42.612203 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0929 22:52:42.612205 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 22:52:42.612535 (+ 330us) server_negotiation.cc:407] Connection header received
0929 22:52:42.612643 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 22:52:42.612644 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 22:52:42.612653 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 22:52:42.612683 (+ 30us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 22:52:42.613899 (+ 1216us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:52:42.614592 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:52:42.615863 (+ 1271us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:52:42.616388 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:52:42.616433 (+ 45us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 22:52:42.616442 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0929 22:52:42.617086 (+ 644us) server_negotiation.cc:299] Negotiation successful
0929 22:52:42.617136 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260929 22:52:42.617259 28199 raft_consensus.cc:2781] T 1188e6d8f5e149a3995fa1b033ecb1c9 P a143941d3e8349a2941b79eb89915ba4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 22:52:42.617297 28199 raft_consensus.cc:482] T 1188e6d8f5e149a3995fa1b033ecb1c9 P a143941d3e8349a2941b79eb89915ba4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 22:52:42.617319 28199 raft_consensus.cc:3037] T 1188e6d8f5e149a3995fa1b033ecb1c9 P a143941d3e8349a2941b79eb89915ba4 [term 0 FOLLOWER]: Advancing to term 1
I20260929 22:52:42.617678 28206 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:52:42.611867 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.147.194:38247 (local address 127.14.147.195:56983)
0929 22:52:42.612429 (+ 562us) negotiation.cc:107] Waiting for socket to connect
0929 22:52:42.612441 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0929 22:52:42.612565 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 22:52:42.612751 (+ 186us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 22:52:42.612751 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 22:52:42.612754 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 22:52:42.612874 (+ 120us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 22:52:42.612876 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 22:52:42.614685 (+ 1809us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 22:52:42.614686 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 22:52:42.615772 (+ 1086us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 22:52:42.615775 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 22:52:42.616785 (+ 1010us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 22:52:42.616785 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 22:52:42.616927 (+ 142us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 22:52:42.616932 (+ 5us) client_negotiation.cc:769] Sending connection context
0929 22:52:42.617561 (+ 629us) client_negotiation.cc:240] Negotiation successful
0929 22:52:42.617606 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":544,"thread_start_us":106,"threads_started":1}
I20260929 22:52:42.618115 27973 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1188e6d8f5e149a3995fa1b033ecb1c9" candidate_uuid: "a143941d3e8349a2941b79eb89915ba4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fcf7eed1fd374116b677881d7eaadbd4" is_pre_election: true
I20260929 22:52:42.618229 28199 raft_consensus.cc:504] T 1188e6d8f5e149a3995fa1b033ecb1c9 P a143941d3e8349a2941b79eb89915ba4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a143941d3e8349a2941b79eb89915ba4" member_type: VOTER last_known_addr { host: "127.14.147.195" port: 41647 } } peers { permanent_uuid: "738ca65869ba46809f58b052437e15e2" member_type: VOTER last_known_addr { host: "127.14.147.193" port: 44211 } } peers { permanent_uuid: "fcf7eed1fd374116b677881d7eaadbd4" member_type: VOTER last_known_addr { host: "127.14.147.194" port: 38247 } }
I20260929 22:52:42.618283 27973 raft_consensus.cc:2454] T 1188e6d8f5e149a3995fa1b033ecb1c9 P fcf7eed1fd374116b677881d7eaadbd4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a143941d3e8349a2941b79eb89915ba4 in term 0.
I20260929 22:52:42.618357 28199 leader_election.cc:288] T 1188e6d8f5e149a3995fa1b033ecb1c9 P a143941d3e8349a2941b79eb89915ba4 [CANDIDATE]: Term 1 election: Requested vote from peers 738ca65869ba46809f58b052437e15e2 (127.14.147.193:44211), fcf7eed1fd374116b677881d7eaadbd4 (127.14.147.194:38247)
I20260929 22:52:42.618681 27973 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1188e6d8f5e149a3995fa1b033ecb1c9" candidate_uuid: "a143941d3e8349a2941b79eb89915ba4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fcf7eed1fd374116b677881d7eaadbd4"
I20260929 22:52:42.618750 27973 raft_consensus.cc:3037] T 1188e6d8f5e149a3995fa1b033ecb1c9 P fcf7eed1fd374116b677881d7eaadbd4 [term 0 FOLLOWER]: Advancing to term 1
I20260929 22:52:42.619773 27973 raft_consensus.cc:2454] T 1188e6d8f5e149a3995fa1b033ecb1c9 P fcf7eed1fd374116b677881d7eaadbd4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a143941d3e8349a2941b79eb89915ba4 in term 1.
I20260929 22:52:42.620080 28048 leader_election.cc:302] T 1188e6d8f5e149a3995fa1b033ecb1c9 P a143941d3e8349a2941b79eb89915ba4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a143941d3e8349a2941b79eb89915ba4, fcf7eed1fd374116b677881d7eaadbd4; no voters:
I20260929 22:52:42.620250 27831 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1188e6d8f5e149a3995fa1b033ecb1c9" candidate_uuid: "a143941d3e8349a2941b79eb89915ba4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "738ca65869ba46809f58b052437e15e2"
I20260929 22:52:42.620358 27831 raft_consensus.cc:3037] T 1188e6d8f5e149a3995fa1b033ecb1c9 P 738ca65869ba46809f58b052437e15e2 [term 0 FOLLOWER]: Advancing to term 1
I20260929 22:52:42.621279 27831 raft_consensus.cc:2454] T 1188e6d8f5e149a3995fa1b033ecb1c9 P 738ca65869ba46809f58b052437e15e2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a143941d3e8349a2941b79eb89915ba4 in term 1.
I20260929 22:52:42.621502 28199 raft_consensus.cc:2781] T 1188e6d8f5e149a3995fa1b033ecb1c9 P a143941d3e8349a2941b79eb89915ba4 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 22:52:42.621675 28199 raft_consensus.cc:686] T 1188e6d8f5e149a3995fa1b033ecb1c9 P a143941d3e8349a2941b79eb89915ba4 [term 1 LEADER]: Becoming Leader. State: Replica: a143941d3e8349a2941b79eb89915ba4, State: Running, Role: LEADER
I20260929 22:52:42.621778 28199 consensus_queue.cc:237] T 1188e6d8f5e149a3995fa1b033ecb1c9 P a143941d3e8349a2941b79eb89915ba4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a143941d3e8349a2941b79eb89915ba4" member_type: VOTER last_known_addr { host: "127.14.147.195" port: 41647 } } peers { permanent_uuid: "738ca65869ba46809f58b052437e15e2" member_type: VOTER last_known_addr { host: "127.14.147.193" port: 44211 } } peers { permanent_uuid: "fcf7eed1fd374116b677881d7eaadbd4" member_type: VOTER last_known_addr { host: "127.14.147.194" port: 38247 } }
I20260929 22:52:42.622650 27693 catalog_manager.cc:5495] T 1188e6d8f5e149a3995fa1b033ecb1c9 P a143941d3e8349a2941b79eb89915ba4 reported cstate change: term changed from 0 to 1, leader changed from <none> to a143941d3e8349a2941b79eb89915ba4 (127.14.147.195). New cstate: current_term: 1 leader_uuid: "a143941d3e8349a2941b79eb89915ba4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a143941d3e8349a2941b79eb89915ba4" member_type: VOTER last_known_addr { host: "127.14.147.195" port: 41647 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "738ca65869ba46809f58b052437e15e2" member_type: VOTER last_known_addr { host: "127.14.147.193" port: 44211 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fcf7eed1fd374116b677881d7eaadbd4" member_type: VOTER last_known_addr { host: "127.14.147.194" port: 38247 } health_report { overall_health: UNKNOWN } } }
I20260929 22:52:42.700542 28191 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:52:42.697956 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56010 (local address 127.14.147.195:41647)
0929 22:52:42.698001 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0929 22:52:42.698003 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 22:52:42.698011 (+ 8us) server_negotiation.cc:407] Connection header received
0929 22:52:42.698084 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 22:52:42.698085 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 22:52:42.698106 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 22:52:42.698168 (+ 62us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 22:52:42.698298 (+ 130us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:52:42.698856 (+ 558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:52:42.699672 (+ 816us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:52:42.700010 (+ 338us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:52:42.700086 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 22:52:42.700196 (+ 110us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 22:52:42.700345 (+ 149us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 22:52:42.700424 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0929 22:52:42.700440 (+ 16us) server_negotiation.cc:299] Negotiation successful
0929 22:52:42.700463 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260929 22:52:42.702564 27831 raft_consensus.cc:1261] T 1188e6d8f5e149a3995fa1b033ecb1c9 P 738ca65869ba46809f58b052437e15e2 [term 1 FOLLOWER]: Refusing update from remote peer a143941d3e8349a2941b79eb89915ba4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 22:52:42.702584 27973 raft_consensus.cc:1261] T 1188e6d8f5e149a3995fa1b033ecb1c9 P fcf7eed1fd374116b677881d7eaadbd4 [term 1 FOLLOWER]: Refusing update from remote peer a143941d3e8349a2941b79eb89915ba4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 22:52:42.702929 28208 consensus_queue.cc:1035] T 1188e6d8f5e149a3995fa1b033ecb1c9 P a143941d3e8349a2941b79eb89915ba4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "738ca65869ba46809f58b052437e15e2" member_type: VOTER last_known_addr { host: "127.14.147.193" port: 44211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 22:52:42.703054 28199 consensus_queue.cc:1035] T 1188e6d8f5e149a3995fa1b033ecb1c9 P a143941d3e8349a2941b79eb89915ba4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fcf7eed1fd374116b677881d7eaadbd4" member_type: VOTER last_known_addr { host: "127.14.147.194" port: 38247 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 22:52:42.705531 28216 mvcc.cc:205] Tried to move back new op lower bound from 7334798797627248640 to 7334798797299400704. Current Snapshot: MvccSnapshot[applied={T|T < 7334798797627248640}]
I20260929 22:52:42.705924 28218 mvcc.cc:205] Tried to move back new op lower bound from 7334798797627248640 to 7334798797299400704. Current Snapshot: MvccSnapshot[applied={T|T < 7334798797627248640}]
I20260929 22:52:42.707211 28217 mvcc.cc:205] Tried to move back new op lower bound from 7334798797627248640 to 7334798797299400704. Current Snapshot: MvccSnapshot[applied={T|T < 7334798797627248640 or (T in {7334798797627248640})}]
I20260929 22:52:42.708958 27693 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41664:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 22:52:42.709066 27693 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41664:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 22:52:42.709901 27693 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 31ac56a028ae4c99bcfe0cde4b1efc91: Sending DeleteTablet for 3 replicas of tablet 1188e6d8f5e149a3995fa1b033ecb1c9
I20260929 22:52:42.710242 28093 tablet_service.cc:1514] Processing DeleteTablet for tablet 1188e6d8f5e149a3995fa1b033ecb1c9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 22:52:42 UTC) from {username='kudu', principal='kudu/127.14.147.254@KRBTEST.COM'} at 127.0.0.1:56004
I20260929 22:52:42.710480 28225 tablet_replica.cc:330] T 1188e6d8f5e149a3995fa1b033ecb1c9 P a143941d3e8349a2941b79eb89915ba4: stopping tablet replica
I20260929 22:52:42.710641 28225 raft_consensus.cc:2229] T 1188e6d8f5e149a3995fa1b033ecb1c9 P a143941d3e8349a2941b79eb89915ba4 [term 1 LEADER]: Raft consensus shutting down.
I20260929 22:52:42.710716 27825 tablet_service.cc:1514] Processing DeleteTablet for tablet 1188e6d8f5e149a3995fa1b033ecb1c9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 22:52:42 UTC) from {username='kudu', principal='kudu/127.14.147.254@KRBTEST.COM'} at 127.0.0.1:60922
I20260929 22:52:42.710884 28225 raft_consensus.cc:2258] T 1188e6d8f5e149a3995fa1b033ecb1c9 P a143941d3e8349a2941b79eb89915ba4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 22:52:42.711233 27941 tablet_service.cc:1514] Processing DeleteTablet for tablet 1188e6d8f5e149a3995fa1b033ecb1c9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 22:52:42 UTC) from {username='kudu', principal='kudu/127.14.147.254@KRBTEST.COM'} at 127.0.0.1:53844
I20260929 22:52:42.711503 28225 ts_tablet_manager.cc:1905] T 1188e6d8f5e149a3995fa1b033ecb1c9 P a143941d3e8349a2941b79eb89915ba4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 22:52:42.711566 28226 tablet_replica.cc:330] T 1188e6d8f5e149a3995fa1b033ecb1c9 P 738ca65869ba46809f58b052437e15e2: stopping tablet replica
I20260929 22:52:42.711664 14927 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhWOCLk/build/release/bin/kudu with pid 27731
I20260929 22:52:42.711678 28226 raft_consensus.cc:2229] T 1188e6d8f5e149a3995fa1b033ecb1c9 P 738ca65869ba46809f58b052437e15e2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 22:52:42.711773 28226 raft_consensus.cc:2258] T 1188e6d8f5e149a3995fa1b033ecb1c9 P 738ca65869ba46809f58b052437e15e2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 22:52:42.711911 28227 tablet_replica.cc:330] T 1188e6d8f5e149a3995fa1b033ecb1c9 P fcf7eed1fd374116b677881d7eaadbd4: stopping tablet replica
I20260929 22:52:42.711995 28227 raft_consensus.cc:2229] T 1188e6d8f5e149a3995fa1b033ecb1c9 P fcf7eed1fd374116b677881d7eaadbd4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 22:52:42.712108 28227 raft_consensus.cc:2258] T 1188e6d8f5e149a3995fa1b033ecb1c9 P fcf7eed1fd374116b677881d7eaadbd4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 22:52:42.714268 28225 ts_tablet_manager.cc:1918] T 1188e6d8f5e149a3995fa1b033ecb1c9 P a143941d3e8349a2941b79eb89915ba4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 22:52:42.714355 28225 log.cc:1196] T 1188e6d8f5e149a3995fa1b033ecb1c9 P a143941d3e8349a2941b79eb89915ba4: Deleting WAL directory at /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790722213368738-14927-0/minicluster-data/ts-2/wal/wals/1188e6d8f5e149a3995fa1b033ecb1c9
I20260929 22:52:42.714666 28225 ts_tablet_manager.cc:1939] T 1188e6d8f5e149a3995fa1b033ecb1c9 P a143941d3e8349a2941b79eb89915ba4: Deleting consensus metadata
I20260929 22:52:42.715129 27678 catalog_manager.cc:4841] TS a143941d3e8349a2941b79eb89915ba4 (127.14.147.195:41647): tablet 1188e6d8f5e149a3995fa1b033ecb1c9 (table test-table [id=d88954e9def14856ab6512e283f80154]) successfully deleted
I20260929 22:52:42.716202 28227 ts_tablet_manager.cc:1905] T 1188e6d8f5e149a3995fa1b033ecb1c9 P fcf7eed1fd374116b677881d7eaadbd4: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260929 22:52:42.717331 27680 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.14.147.193:44211 (error 108)
I20260929 22:52:42.718007 14927 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhWOCLk/build/release/bin/kudu with pid 27895
I20260929 22:52:42.718029 28227 ts_tablet_manager.cc:1918] T 1188e6d8f5e149a3995fa1b033ecb1c9 P fcf7eed1fd374116b677881d7eaadbd4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 22:52:42.718212 28227 log.cc:1196] T 1188e6d8f5e149a3995fa1b033ecb1c9 P fcf7eed1fd374116b677881d7eaadbd4: Deleting WAL directory at /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790722213368738-14927-0/minicluster-data/ts-1/wal/wals/1188e6d8f5e149a3995fa1b033ecb1c9
I20260929 22:52:42.721928 28187 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:52:42.721771 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.147.193:44211 (local address 127.0.0.1:60928)
0929 22:52:42.721841 (+ 70us) negotiation.cc:107] Waiting for socket to connect
0929 22:52:42.721881 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.147.193:44211: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260929 22:52:42.722078 27680 catalog_manager.cc:4584] TS 738ca65869ba46809f58b052437e15e2 (127.14.147.193:44211): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 1188e6d8f5e149a3995fa1b033ecb1c9: Network error: Client connection negotiation failed: client connection to 127.14.147.193:44211: connect: Connection refused (error 111)
I20260929 22:52:42.723513 28187 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:52:42.723380 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.147.194:38247 (local address 127.0.0.1:53856)
0929 22:52:42.723451 (+ 71us) negotiation.cc:107] Waiting for socket to connect
0929 22:52:42.723477 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.147.194:38247: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
I20260929 22:52:42.723661 14927 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhWOCLk/build/release/bin/kudu with pid 28028
I20260929 22:52:42.730384 14927 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhWOCLk/build/release/bin/kudu with pid 27660
I20260929 22:52:42.737035 14927 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260929 22:52:43.072209 14927 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260929 22:52:43.072324 14927 mini_ranger.cc:66] Stopping Ranger...
I20260929 22:52:43.407260 14927 mini_ranger.cc:68] Stopped Ranger
2026-09-29 22:52:43.407 UTC [26727] LOG: received smart shutdown request
2026-09-29 22:52:43.412 UTC [26727] LOG: background worker "logical replication launcher" (PID 26736) exited with exit code 1
2026-09-29 22:52:43.412 UTC [26731] LOG: shutting down
2026-09-29 22:52:43.561 UTC [26727] LOG: database system is shut down
2026-09-29T22:52:43Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers (48799 ms)
[ RUN ] SecurityITest.IPKICACert
Loading random data
Initializing database '/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790722213368738-14927-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 22:52:43 dist-test-slave-98mk krb5kdc[28237](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 22:52:43 dist-test-slave-98mk krb5kdc[28237](info): set up 2 sockets
Sep 29 22:52:43 dist-test-slave-98mk krb5kdc[28237](info): commencing operation
krb5kdc: starting...
W20260929 22:52:44.603814 14927 mini_kdc.cc:121] Time spent starting KDC: real 0.931s user 0.002s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790722213368738-14927-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790722213368738-14927-0/krb5kdc/test-user.keytab.
Sep 29 22:52:44 dist-test-slave-98mk krb5kdc[28237](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790722364, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T22:52:44Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T22:52:44Z Disabled control of system clock
WARNING: no policy specified for kudu/127.14.147.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.147.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.147.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790722213368738-14927-0/krb5kdc/kudu_127.14.147.254.keytab.
Entry for principal kudu/127.14.147.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790722213368738-14927-0/krb5kdc/kudu_127.14.147.254.keytab.
I20260929 22:52:44.716355 14927 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhWOCLk/build/release/bin/kudu
/tmp/dist-test-taskhWOCLk/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790722213368738-14927-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790722213368738-14927-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790722213368738-14927-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790722213368738-14927-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.147.254:33093
--webserver_interface=127.14.147.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790722213368738-14927-0/krb5kdc/kudu_127.14.147.254.keytab
--principal=kudu/127.14.147.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.147.212:45487
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.147.254:33093,127.14.147.253:40371,127.14.147.252:33573
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790722213368738-14927-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790722213368738-14927-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790722213368738-14927-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 22:52:44.796658 28250 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 22:52:44.796885 28250 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 22:52:44.796916 28250 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 22:52:44.798058 28250 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 22:52:44.798100 28250 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 22:52:44.798118 28250 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 22:52:44.798134 28250 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 22:52:44.798149 28250 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 22:52:44.799669 28250 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.147.212:45487
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790722213368738-14927-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790722213368738-14927-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.14.147.254:33093,127.14.147.253:40371,127.14.147.252:33573
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790722213368738-14927-0/krb5kdc/kudu_127.14.147.254.keytab
--principal=kudu/127.14.147.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.147.254:33093
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790722213368738-14927-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.14.147.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28250
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790722213368738-14927-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 21:58:54 UTC on ccycloud.quasar-xguoti.root.comops.site
build id 4681390
I20260929 22:52:44.799942 28250 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 22:52:44.800153 28250 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 22:52:44.804768 28256 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 22:52:44.804775 28258 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 22:52:44.804775 28255 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 22:52:44.805028 28250 server_base.cc:1034] running on GCE node
I20260929 22:52:44.805279 28250 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 22:52:44.805586 28250 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 22:52:44.806736 28250 hybrid_clock.cc:648] HybridClock initialized: now 1790722364806723 us; error 35 us; skew 500 ppm
Sep 29 22:52:44 dist-test-slave-98mk krb5kdc[28237](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790722364, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.147.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 22:52:44.812157 28250 init.cc:377] Logged in from keytab as kudu/127.14.147.254@KRBTEST.COM (short username kudu)
I20260929 22:52:44.812599 28250 webserver.cc:460] Webserver started at http://127.14.147.254:39773/ using document root <none> and password file <none>
I20260929 22:52:44.812812 28250 fs_manager.cc:359] Metadata directory not provided
I20260929 22:52:44.812861 28250 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 22:52:44.812987 28250 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 22:52:44.814066 28250 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790722213368738-14927-0/minicluster-data/master-0/data/instance:
uuid: "1b35099a272c43798770ed452b8c6eab"
format_stamp: "Formatted at 2026-09-29 22:52:44 on dist-test-slave-98mk"
server_key: "10c5024d81ff88b6563308827695f51f"
server_key_iv: "8e1a5040b5c3555c8d7cbe2cdd2e27a2"
server_key_version: "encryptionkey@0"
I20260929 22:52:44.814415 28250 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790722213368738-14927-0/minicluster-data/master-0/wal/instance:
uuid: "1b35099a272c43798770ed452b8c6eab"
format_stamp: "Formatted at 2026-09-29 22:52:44 on dist-test-slave-98mk"
server_key: "10c5024d81ff88b6563308827695f51f"
server_key_iv: "8e1a5040b5c3555c8d7cbe2cdd2e27a2"
server_key_version: "encryptionkey@0"
I20260929 22:52:44.815781 28250 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260929 22:52:44.816546 28265 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 22:52:44.816730 28250 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260929 22:52:44.816818 28250 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790722213368738-14927-0/minicluster-data/master-0/data,/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790722213368738-14927-0/minicluster-data/master-0/wal
uuid: "1b35099a272c43798770ed452b8c6eab"
format_stamp: "Formatted at 2026-09-29 22:52:44 on dist-test-slave-98mk"
server_key: "10c5024d81ff88b6563308827695f51f"
server_key_iv: "8e1a5040b5c3555c8d7cbe2cdd2e27a2"
server_key_version: "encryptionkey@0"
I20260929 22:52:44.816881 28250 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790722213368738-14927-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790722213368738-14927-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790722213368738-14927-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 22:52:44.829020 28250 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 22:52:44.831149 28250 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 22:52:44.831296 28250 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 22:52:44.835166 28250 rpc_server.cc:307] RPC server started. Bound to: 127.14.147.254:33093
I20260929 22:52:44.835191 28317 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.147.254:33093 every 8 connection(s)
I20260929 22:52:44.835584 28250 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790722213368738-14927-0/minicluster-data/master-0/data/info.pb
I20260929 22:52:44.836480 28318 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 22:52:44.838020 28318 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.14.147.254" port: 33093 } has no permanent_uuid. Determining permanent_uuid...
I20260929 22:52:44.841272 14927 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhWOCLk/build/release/bin/kudu as pid 28250
I20260929 22:52:44.841377 14927 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790722213368738-14927-0/minicluster-data/master-0/wal/instance
I20260929 22:52:44.841544 14927 external_mini_cluster.cc:1428] Setting key 3aef2867abd5a29c7c1922a85cbfdf35
Sep 29 22:52:44 dist-test-slave-98mk krb5kdc[28237](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790722364, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.147.254@KRBTEST.COM for kudu/127.14.147.254@KRBTEST.COM
I20260929 22:52:44.848222 28321 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:52:44.838723 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43728 (local address 127.14.147.254:33093)
0929 22:52:44.838893 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0929 22:52:44.838903 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 22:52:44.838978 (+ 75us) server_negotiation.cc:407] Connection header received
0929 22:52:44.839493 (+ 515us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 22:52:44.839495 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 22:52:44.839514 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 22:52:44.839596 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0929 22:52:44.840225 (+ 629us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:52:44.841394 (+ 1169us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:52:44.842356 (+ 962us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:52:44.842688 (+ 332us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:52:44.842799 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 22:52:44.845076 (+ 2277us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 22:52:44.845084 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 22:52:44.845085 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 22:52:44.845092 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0929 22:52:44.846593 (+ 1501us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 22:52:44.847037 (+ 444us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 22:52:44.847039 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 22:52:44.847039 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 22:52:44.847115 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 22:52:44.847347 (+ 232us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 22:52:44.847347 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 22:52:44.847348 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 22:52:44.847617 (+ 269us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 22:52:44.847682 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0929 22:52:44.847896 (+ 214us) server_negotiation.cc:299] Negotiation successful
0929 22:52:44.848011 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":127,"thread_start_us":80,"threads_started":1}
I20260929 22:52:44.848227 28320 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:52:44.838695 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.147.254:33093 (local address 127.0.0.1:43728)
0929 22:52:44.838852 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0929 22:52:44.838893 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0929 22:52:44.839345 (+ 452us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 22:52:44.839610 (+ 265us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 22:52:44.839613 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 22:52:44.839863 (+ 250us) client_negotiation.cc:189] Negotiated authn=SASL
0929 22:52:44.840031 (+ 168us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 22:52:44.840035 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 22:52:44.841503 (+ 1468us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 22:52:44.841504 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 22:52:44.842270 (+ 766us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 22:52:44.842273 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 22:52:44.842767 (+ 494us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 22:52:44.842768 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 22:52:44.842831 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 22:52:44.843345 (+ 514us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 22:52:44.843353 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0929 22:52:44.844964 (+ 1611us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 22:52:44.846686 (+ 1722us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 22:52:44.846695 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 22:52:44.846697 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 22:52:44.846963 (+ 266us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 22:52:44.847173 (+ 210us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 22:52:44.847174 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 22:52:44.847174 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 22:52:44.847290 (+ 116us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 22:52:44.847682 (+ 392us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 22:52:44.847689 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 22:52:44.847770 (+ 81us) client_negotiation.cc:769] Sending connection context
0929 22:52:44.847879 (+ 109us) client_negotiation.cc:240] Negotiation successful
0929 22:52:44.848087 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":115,"mutex_wait_us":45,"thread_start_us":71,"threads_started":1}
WARNING: no policy specified for kudu/127.14.147.253@KRBTEST.COM; defaulting to no policy
I20260929 22:52:44.849027 28318 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.14.147.253" port: 40371 } has no permanent_uuid. Determining permanent_uuid...
I20260929 22:52:44.849370 28320 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:52:44.849259 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.147.253:40371 (local address 127.0.0.1:56014)
0929 22:52:44.849305 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0929 22:52:44.849347 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.147.253:40371: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":28}
W20260929 22:52:44.849501 28266 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.14.147.253:40371: connect: Connection refused (error 111)
I20260929 22:52:44.849926 28320 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:52:44.849824 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.147.253:40371 (local address 127.0.0.1:56024)
0929 22:52:44.849867 (+ 43us) negotiation.cc:107] Waiting for socket to connect
0929 22:52:44.849896 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.147.253:40371: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":28}
W20260929 22:52:44.850083 28318 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.14.147.253:40371: Network error: Client connection negotiation failed: client connection to 127.14.147.253:40371: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.147.253@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.147.253 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790722213368738-14927-0/krb5kdc/kudu_127.14.147.253.keytab.
Entry for principal kudu/127.14.147.253 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790722213368738-14927-0/krb5kdc/kudu_127.14.147.253.keytab.
I20260929 22:52:44.863911 14927 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhWOCLk/build/release/bin/kudu
/tmp/dist-test-taskhWOCLk/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790722213368738-14927-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790722213368738-14927-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790722213368738-14927-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790722213368738-14927-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.147.253:40371
--webserver_interface=127.14.147.253
--webserver_port=0
--keytab_file=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790722213368738-14927-0/krb5kdc/kudu_127.14.147.253.keytab
--principal=kudu/127.14.147.253
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.147.212:45487
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.147.254:33093,127.14.147.253:40371,127.14.147.252:33573
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790722213368738-14927-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790722213368738-14927-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790722213368738-14927-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 22:52:44.899360 28318 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.14.147.253" port: 40371 } attempt: 1
I20260929 22:52:44.900077 28320 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:52:44.899784 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.147.253:40371 (local address 127.0.0.1:56036)
0929 22:52:44.899935 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0929 22:52:44.900020 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.147.253:40371: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":101}
I20260929 22:52:44.900624 28320 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:52:44.900516 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.147.253:40371 (local address 127.0.0.1:56050)
0929 22:52:44.900565 (+ 49us) negotiation.cc:107] Waiting for socket to connect
0929 22:52:44.900598 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.147.253:40371: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
W20260929 22:52:44.900761 28318 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.14.147.253:40371: Network error: Client connection negotiation failed: client connection to 127.14.147.253:40371: connect: Connection refused (error 111)
W20260929 22:52:44.944700 28325 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 22:52:44.944934 28325 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 22:52:44.944957 28325 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 22:52:44.946161 28325 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 22:52:44.946208 28325 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 22:52:44.946221 28325 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 22:52:44.946233 28325 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 22:52:44.946244 28325 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 22:52:44.947837 28325 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.147.212:45487
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790722213368738-14927-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790722213368738-14927-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.14.147.254:33093,127.14.147.253:40371,127.14.147.252:33573
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790722213368738-14927-0/krb5kdc/kudu_127.14.147.253.keytab
--principal=kudu/127.14.147.253
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.147.253:40371
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790722213368738-14927-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.14.147.253
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28325
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790722213368738-14927-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 21:58:54 UTC on ccycloud.quasar-xguoti.root.comops.site
build id 4681390
I20260929 22:52:44.948081 28325 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 22:52:44.948304 28325 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 22:52:44.952973 28331 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 22:52:44.952993 28333 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 22:52:44.952993 28330 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 22:52:44.953200 28325 server_base.cc:1034] running on GCE node
I20260929 22:52:44.953431 28325 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 22:52:44.953672 28325 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 22:52:44.954818 28325 hybrid_clock.cc:648] HybridClock initialized: now 1790722364954803 us; error 28 us; skew 500 ppm
Sep 29 22:52:44 dist-test-slave-98mk krb5kdc[28237](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790722364, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.147.253@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 22:52:44.957206 28325 init.cc:377] Logged in from keytab as kudu/127.14.147.253@KRBTEST.COM (short username kudu)
I20260929 22:52:44.957638 28325 webserver.cc:460] Webserver started at http://127.14.147.253:39031/ using document root <none> and password file <none>
I20260929 22:52:44.957857 28325 fs_manager.cc:359] Metadata directory not provided
I20260929 22:52:44.957904 28325 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 22:52:44.958025 28325 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 22:52:44.959090 28325 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790722213368738-14927-0/minicluster-data/master-1/data/instance:
uuid: "303754d07d40449c97a83aede259ba4d"
format_stamp: "Formatted at 2026-09-29 22:52:44 on dist-test-slave-98mk"
server_key: "1d5015e97f629265ba27aae3e51fe0e3"
server_key_iv: "6e75670d8d4b4ac51ed372ba9a63f26f"
server_key_version: "encryptionkey@0"
I20260929 22:52:44.959444 28325 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790722213368738-14927-0/minicluster-data/master-1/wal/instance:
uuid: "303754d07d40449c97a83aede259ba4d"
format_stamp: "Formatted at 2026-09-29 22:52:44 on dist-test-slave-98mk"
server_key: "1d5015e97f629265ba27aae3e51fe0e3"
server_key_iv: "6e75670d8d4b4ac51ed372ba9a63f26f"
server_key_version: "encryptionkey@0"
I20260929 22:52:44.960680 28325 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260929 22:52:44.961378 28340 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 22:52:44.961544 28325 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260929 22:52:44.961624 28325 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790722213368738-14927-0/minicluster-data/master-1/data,/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790722213368738-14927-0/minicluster-data/master-1/wal
uuid: "303754d07d40449c97a83aede259ba4d"
format_stamp: "Formatted at 2026-09-29 22:52:44 on dist-test-slave-98mk"
server_key: "1d5015e97f629265ba27aae3e51fe0e3"
server_key_iv: "6e75670d8d4b4ac51ed372ba9a63f26f"
server_key_version: "encryptionkey@0"
I20260929 22:52:44.961690 28325 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790722213368738-14927-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790722213368738-14927-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790722213368738-14927-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 22:52:44.968935 28318 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.14.147.253" port: 40371 } attempt: 2
I20260929 22:52:44.969373 28320 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:52:44.969253 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.147.253:40371 (local address 127.0.0.1:56064)
0929 22:52:44.969311 (+ 58us) negotiation.cc:107] Waiting for socket to connect
0929 22:52:44.969343 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.147.253:40371: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
I20260929 22:52:44.969666 28320 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:52:44.969588 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.147.253:40371 (local address 127.0.0.1:56080)
0929 22:52:44.969623 (+ 35us) negotiation.cc:107] Waiting for socket to connect
0929 22:52:44.969646 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.147.253:40371: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":21}
W20260929 22:52:44.969741 28318 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.14.147.253:40371: Network error: Client connection negotiation failed: client connection to 127.14.147.253:40371: connect: Connection refused (error 111)
I20260929 22:52:44.972224 28325 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 22:52:44.974296 28325 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 22:52:44.974429 28325 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 22:52:44.978857 28325 rpc_server.cc:307] RPC server started. Bound to: 127.14.147.253:40371
I20260929 22:52:44.978883 28392 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.147.253:40371 every 8 connection(s)
I20260929 22:52:44.979364 28325 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790722213368738-14927-0/minicluster-data/master-1/data/info.pb
I20260929 22:52:44.980015 28393 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 22:52:44.981503 28393 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.14.147.254" port: 33093 } has no permanent_uuid. Determining permanent_uuid...
Sep 29 22:52:44 dist-test-slave-98mk krb5kdc[28237](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790722364, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.147.253@KRBTEST.COM for kudu/127.14.147.254@KRBTEST.COM
I20260929 22:52:44.988781 14927 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhWOCLk/build/release/bin/kudu as pid 28325
I20260929 22:52:44.988895 14927 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790722213368738-14927-0/minicluster-data/master-1/wal/instance
I20260929 22:52:44.989061 14927 external_mini_cluster.cc:1428] Setting key 377a3fc35548b84f900d80c9cf35cac9
I20260929 22:52:44.990060 28321 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:52:44.982180 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43744 (local address 127.14.147.254:33093)
0929 22:52:44.982257 (+ 77us) server_negotiation.cc:206] Beginning negotiation
0929 22:52:44.982260 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 22:52:44.982407 (+ 147us) server_negotiation.cc:407] Connection header received
0929 22:52:44.982843 (+ 436us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 22:52:44.982844 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 22:52:44.982853 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 22:52:44.982902 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0929 22:52:44.983492 (+ 590us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:52:44.983981 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:52:44.984969 (+ 988us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:52:44.985175 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:52:44.985230 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 22:52:44.987358 (+ 2128us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 22:52:44.987360 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 22:52:44.987360 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 22:52:44.987367 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0929 22:52:44.988714 (+ 1347us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 22:52:44.989131 (+ 417us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 22:52:44.989133 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 22:52:44.989133 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 22:52:44.989192 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 22:52:44.989433 (+ 241us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 22:52:44.989433 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 22:52:44.989434 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 22:52:44.989551 (+ 117us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 22:52:44.989607 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0929 22:52:44.989879 (+ 272us) server_negotiation.cc:299] Negotiation successful
0929 22:52:44.989969 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260929 22:52:44.990258 28395 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:52:44.982136 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.147.254:33093 (local address 127.0.0.1:43744)
0929 22:52:44.982284 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0929 22:52:44.982328 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0929 22:52:44.982769 (+ 441us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 22:52:44.982955 (+ 186us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 22:52:44.982958 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 22:52:44.983194 (+ 236us) client_negotiation.cc:189] Negotiated authn=SASL
0929 22:52:44.983394 (+ 200us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 22:52:44.983398 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 22:52:44.984042 (+ 644us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 22:52:44.984042 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 22:52:44.984886 (+ 844us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 22:52:44.984889 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 22:52:44.985259 (+ 370us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 22:52:44.985260 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 22:52:44.985384 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 22:52:44.985899 (+ 515us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 22:52:44.985908 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0929 22:52:44.987267 (+ 1359us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 22:52:44.988794 (+ 1527us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 22:52:44.988799 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 22:52:44.988801 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 22:52:44.989063 (+ 262us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 22:52:44.989250 (+ 187us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 22:52:44.989250 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 22:52:44.989251 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 22:52:44.989374 (+ 123us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 22:52:44.989605 (+ 231us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 22:52:44.989612 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 22:52:44.989752 (+ 140us) client_negotiation.cc:769] Sending connection context
0929 22:52:44.989844 (+ 92us) client_negotiation.cc:240] Negotiation successful
0929 22:52:44.990010 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":72,"threads_started":1}
I20260929 22:52:44.990845 28393 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.14.147.253" port: 40371 } has no permanent_uuid. Determining permanent_uuid...
Sep 29 22:52:44 dist-test-slave-98mk krb5kdc[28237](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790722364, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.147.253@KRBTEST.COM for kudu/127.14.147.253@KRBTEST.COM
WARNING: no policy specified for kudu/127.14.147.252@KRBTEST.COM; defaulting to no policy
I20260929 22:52:44.999049 28395 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:52:44.991117 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.147.253:40371 (local address 127.0.0.1:56082)
0929 22:52:44.991174 (+ 57us) negotiation.cc:107] Waiting for socket to connect
0929 22:52:44.991185 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0929 22:52:44.991228 (+ 43us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 22:52:44.991510 (+ 282us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 22:52:44.991511 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 22:52:44.991623 (+ 112us) client_negotiation.cc:189] Negotiated authn=SASL
0929 22:52:44.991676 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 22:52:44.991677 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 22:52:44.992890 (+ 1213us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 22:52:44.992891 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 22:52:44.993386 (+ 495us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 22:52:44.993388 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 22:52:44.994057 (+ 669us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 22:52:44.994059 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 22:52:44.994129 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 22:52:44.994577 (+ 448us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 22:52:44.994581 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0929 22:52:44.995977 (+ 1396us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 22:52:44.997645 (+ 1668us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 22:52:44.997646 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 22:52:44.997647 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 22:52:44.997821 (+ 174us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 22:52:44.998124 (+ 303us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 22:52:44.998126 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 22:52:44.998127 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 22:52:44.998243 (+ 116us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 22:52:44.998609 (+ 366us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 22:52:44.998611 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 22:52:44.998670 (+ 59us) client_negotiation.cc:769] Sending connection context
0929 22:52:44.998735 (+ 65us) client_negotiation.cc:240] Negotiation successful
0929 22:52:44.998883 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":41}
I20260929 22:52:44.999094 28397 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:52:44.991172 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56082 (local address 127.14.147.253:40371)
0929 22:52:44.991343 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0929 22:52:44.991347 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 22:52:44.991374 (+ 27us) server_negotiation.cc:407] Connection header received
0929 22:52:44.991395 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 22:52:44.991397 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 22:52:44.991435 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 22:52:44.991501 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0929 22:52:44.991732 (+ 231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:52:44.992787 (+ 1055us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:52:44.993466 (+ 679us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:52:44.993951 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:52:44.994031 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 22:52:44.996072 (+ 2041us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 22:52:44.996083 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 22:52:44.996083 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 22:52:44.996092 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0929 22:52:44.997563 (+ 1471us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 22:52:44.997914 (+ 351us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 22:52:44.997917 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 22:52:44.997918 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 22:52:44.997977 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 22:52:44.998341 (+ 364us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 22:52:44.998341 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 22:52:44.998342 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 22:52:44.998528 (+ 186us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 22:52:44.998603 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0929 22:52:44.998776 (+ 173us) server_negotiation.cc:299] Negotiation successful
0929 22:52:44.998968 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":82,"server-negotiator.queue_time_us":134,"thread_start_us":92,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.147.252@KRBTEST.COM" created.
I20260929 22:52:45.000137 28393 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.14.147.252" port: 33573 } has no permanent_uuid. Determining permanent_uuid...
I20260929 22:52:45.000617 28395 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:52:45.000432 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.147.252:33573 (local address 127.0.0.1:52676)
0929 22:52:45.000503 (+ 71us) negotiation.cc:107] Waiting for socket to connect
0929 22:52:45.000576 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.147.252:33573: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260929 22:52:45.000780 28341 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.14.147.252:33573: connect: Connection refused (error 111)
I20260929 22:52:45.001204 28395 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:52:45.001118 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.147.252:33573 (local address 127.0.0.1:52690)
0929 22:52:45.001155 (+ 37us) negotiation.cc:107] Waiting for socket to connect
0929 22:52:45.001180 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.147.252:33573: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":24}
W20260929 22:52:45.001312 28393 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.14.147.252:33573: Network error: Client connection negotiation failed: client connection to 127.14.147.252:33573: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.147.252 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790722213368738-14927-0/krb5kdc/kudu_127.14.147.252.keytab.
Entry for principal kudu/127.14.147.252 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790722213368738-14927-0/krb5kdc/kudu_127.14.147.252.keytab.
I20260929 22:52:45.014469 14927 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhWOCLk/build/release/bin/kudu
/tmp/dist-test-taskhWOCLk/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790722213368738-14927-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790722213368738-14927-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790722213368738-14927-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790722213368738-14927-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.147.252:33573
--webserver_interface=127.14.147.252
--webserver_port=0
--keytab_file=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790722213368738-14927-0/krb5kdc/kudu_127.14.147.252.keytab
--principal=kudu/127.14.147.252
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.147.212:45487
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.147.254:33093,127.14.147.253:40371,127.14.147.252:33573
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790722213368738-14927-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790722213368738-14927-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790722213368738-14927-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 22:52:45.050556 28393 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.14.147.252" port: 33573 } attempt: 1
I20260929 22:52:45.051156 28395 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:52:45.050990 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.147.252:33573 (local address 127.0.0.1:52700)
0929 22:52:45.051079 (+ 89us) negotiation.cc:107] Waiting for socket to connect
0929 22:52:45.051123 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.147.252:33573: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
I20260929 22:52:45.051555 28395 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:52:45.051482 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.147.252:33573 (local address 127.0.0.1:52706)
0929 22:52:45.051512 (+ 30us) negotiation.cc:107] Waiting for socket to connect
0929 22:52:45.051537 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.147.252:33573: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":16}
W20260929 22:52:45.051643 28393 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.14.147.252:33573: Network error: Client connection negotiation failed: client connection to 127.14.147.252:33573: connect: Connection refused (error 111)
I20260929 22:52:45.060879 28318 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.14.147.253" port: 40371 } attempt: 3
Sep 29 22:52:45 dist-test-slave-98mk krb5kdc[28237](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790722364, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.147.254@KRBTEST.COM for kudu/127.14.147.253@KRBTEST.COM
I20260929 22:52:45.068538 28397 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:52:45.061318 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56086 (local address 127.14.147.253:40371)
0929 22:52:45.061377 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0929 22:52:45.061379 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 22:52:45.061392 (+ 13us) server_negotiation.cc:407] Connection header received
0929 22:52:45.061421 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 22:52:45.061422 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 22:52:45.061435 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 22:52:45.061510 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0929 22:52:45.061790 (+ 280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:52:45.062577 (+ 787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:52:45.063252 (+ 675us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:52:45.063592 (+ 340us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:52:45.063667 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 22:52:45.065565 (+ 1898us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 22:52:45.065567 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 22:52:45.065568 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 22:52:45.065576 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0929 22:52:45.067059 (+ 1483us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 22:52:45.067401 (+ 342us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 22:52:45.067402 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 22:52:45.067403 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 22:52:45.067489 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 22:52:45.067731 (+ 242us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 22:52:45.067732 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 22:52:45.067732 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 22:52:45.067908 (+ 176us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 22:52:45.067993 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0929 22:52:45.068332 (+ 339us) server_negotiation.cc:299] Negotiation successful
0929 22:52:45.068432 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260929 22:52:45.068600 28320 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:52:45.061228 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.147.253:40371 (local address 127.0.0.1:56086)
0929 22:52:45.061280 (+ 52us) negotiation.cc:107] Waiting for socket to connect
0929 22:52:45.061293 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 22:52:45.061341 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 22:52:45.061515 (+ 174us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 22:52:45.061515 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 22:52:45.061660 (+ 145us) client_negotiation.cc:189] Negotiated authn=SASL
0929 22:52:45.061727 (+ 67us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 22:52:45.061728 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 22:52:45.062678 (+ 950us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 22:52:45.062679 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 22:52:45.063180 (+ 501us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 22:52:45.063181 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 22:52:45.063684 (+ 503us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 22:52:45.063685 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 22:52:45.063770 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 22:52:45.064245 (+ 475us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 22:52:45.064250 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0929 22:52:45.065474 (+ 1224us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 22:52:45.067153 (+ 1679us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 22:52:45.067154 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 22:52:45.067155 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 22:52:45.067335 (+ 180us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 22:52:45.067545 (+ 210us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 22:52:45.067546 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 22:52:45.067546 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 22:52:45.067603 (+ 57us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 22:52:45.068080 (+ 477us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 22:52:45.068083 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 22:52:45.068169 (+ 86us) client_negotiation.cc:769] Sending connection context
0929 22:52:45.068281 (+ 112us) client_negotiation.cc:240] Negotiation successful
0929 22:52:45.068487 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":29}
I20260929 22:52:45.069180 28318 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.14.147.252" port: 33573 } has no permanent_uuid. Determining permanent_uuid...
I20260929 22:52:45.069674 28320 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:52:45.069469 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.147.252:33573 (local address 127.0.0.1:52722)
0929 22:52:45.069518 (+ 49us) negotiation.cc:107] Waiting for socket to connect
0929 22:52:45.069629 (+ 111us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.147.252:33573: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
I20260929 22:52:45.070104 28320 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:52:45.070009 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.147.252:33573 (local address 127.0.0.1:52734)
0929 22:52:45.070058 (+ 49us) negotiation.cc:107] Waiting for socket to connect
0929 22:52:45.070082 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.147.252:33573: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
W20260929 22:52:45.070194 28318 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.14.147.252:33573: Network error: Client connection negotiation failed: client connection to 127.14.147.252:33573: connect: Connection refused (error 111)
W20260929 22:52:45.099076 28400 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 22:52:45.099303 28400 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 22:52:45.099334 28400 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 22:52:45.100615 28400 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 22:52:45.100666 28400 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 22:52:45.100685 28400 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 22:52:45.100701 28400 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 22:52:45.100716 28400 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 22:52:45.101351 28318 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.14.147.252" port: 33573 } attempt: 1
I20260929 22:52:45.101794 28320 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:52:45.101651 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.147.252:33573 (local address 127.0.0.1:52746)
0929 22:52:45.101733 (+ 82us) negotiation.cc:107] Waiting for socket to connect
0929 22:52:45.101764 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.147.252:33573: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
I20260929 22:52:45.102108 28320 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:52:45.102027 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.147.252:33573 (local address 127.0.0.1:52762)
0929 22:52:45.102067 (+ 40us) negotiation.cc:107] Waiting for socket to connect
0929 22:52:45.102090 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.147.252:33573: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":25}
W20260929 22:52:45.102190 28318 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.14.147.252:33573: Network error: Client connection negotiation failed: client connection to 127.14.147.252:33573: connect: Connection refused (error 111)
I20260929 22:52:45.102434 28400 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.147.212:45487
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790722213368738-14927-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790722213368738-14927-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.14.147.254:33093,127.14.147.253:40371,127.14.147.252:33573
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790722213368738-14927-0/krb5kdc/kudu_127.14.147.252.keytab
--principal=kudu/127.14.147.252
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.147.252:33573
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790722213368738-14927-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.14.147.252
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28400
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790722213368738-14927-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 21:58:54 UTC on ccycloud.quasar-xguoti.root.comops.site
build id 4681390
I20260929 22:52:45.102684 28400 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 22:52:45.102895 28400 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 22:52:45.107461 28405 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 22:52:45.107463 28406 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 22:52:45.107509 28408 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 22:52:45.107640 28400 server_base.cc:1034] running on GCE node
I20260929 22:52:45.107954 28400 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 22:52:45.108245 28400 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 22:52:45.109391 28400 hybrid_clock.cc:648] HybridClock initialized: now 1790722365109378 us; error 28 us; skew 500 ppm
Sep 29 22:52:45 dist-test-slave-98mk krb5kdc[28237](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790722365, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.147.252@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 22:52:45.111788 28400 init.cc:377] Logged in from keytab as kudu/127.14.147.252@KRBTEST.COM (short username kudu)
I20260929 22:52:45.112211 28400 webserver.cc:460] Webserver started at http://127.14.147.252:38277/ using document root <none> and password file <none>
I20260929 22:52:45.112411 28400 fs_manager.cc:359] Metadata directory not provided
I20260929 22:52:45.112453 28400 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 22:52:45.112545 28400 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 22:52:45.113587 28400 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790722213368738-14927-0/minicluster-data/master-2/data/instance:
uuid: "2d10da5a5bf3463694927f4485c8d83e"
format_stamp: "Formatted at 2026-09-29 22:52:45 on dist-test-slave-98mk"
server_key: "b14e6c69bf40f16ad15f216177efe492"
server_key_iv: "91b8779c38bb39b1ccdd290cff5ba5db"
server_key_version: "encryptionkey@0"
I20260929 22:52:45.113893 28400 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790722213368738-14927-0/minicluster-data/master-2/wal/instance:
uuid: "2d10da5a5bf3463694927f4485c8d83e"
format_stamp: "Formatted at 2026-09-29 22:52:45 on dist-test-slave-98mk"
server_key: "b14e6c69bf40f16ad15f216177efe492"
server_key_iv: "91b8779c38bb39b1ccdd290cff5ba5db"
server_key_version: "encryptionkey@0"
I20260929 22:52:45.115154 28400 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260929 22:52:45.115900 28415 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 22:52:45.116055 28400 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260929 22:52:45.116138 28400 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790722213368738-14927-0/minicluster-data/master-2/data,/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790722213368738-14927-0/minicluster-data/master-2/wal
uuid: "2d10da5a5bf3463694927f4485c8d83e"
format_stamp: "Formatted at 2026-09-29 22:52:45 on dist-test-slave-98mk"
server_key: "b14e6c69bf40f16ad15f216177efe492"
server_key_iv: "91b8779c38bb39b1ccdd290cff5ba5db"
server_key_version: "encryptionkey@0"
I20260929 22:52:45.116204 28400 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790722213368738-14927-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790722213368738-14927-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790722213368738-14927-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 22:52:45.119822 28393 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.14.147.252" port: 33573 } attempt: 2
I20260929 22:52:45.120275 28395 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:52:45.120119 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.147.252:33573 (local address 127.0.0.1:52778)
0929 22:52:45.120175 (+ 56us) negotiation.cc:107] Waiting for socket to connect
0929 22:52:45.120238 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.147.252:33573: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
I20260929 22:52:45.120600 28395 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:52:45.120506 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.147.252:33573 (local address 127.0.0.1:52786)
0929 22:52:45.120546 (+ 40us) negotiation.cc:107] Waiting for socket to connect
0929 22:52:45.120573 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.147.252:33573: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":25}
W20260929 22:52:45.120697 28393 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.14.147.252:33573: Network error: Client connection negotiation failed: client connection to 127.14.147.252:33573: connect: Connection refused (error 111)
I20260929 22:52:45.124154 28400 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 22:52:45.125985 28400 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 22:52:45.126101 28400 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 22:52:45.129992 28400 rpc_server.cc:307] RPC server started. Bound to: 127.14.147.252:33573
I20260929 22:52:45.130030 28467 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.147.252:33573 every 8 connection(s)
I20260929 22:52:45.130376 28400 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790722213368738-14927-0/minicluster-data/master-2/data/info.pb
I20260929 22:52:45.131302 28468 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 22:52:45.133093 28468 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.14.147.254" port: 33093 } has no permanent_uuid. Determining permanent_uuid...
Sep 29 22:52:45 dist-test-slave-98mk krb5kdc[28237](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790722365, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.147.252@KRBTEST.COM for kudu/127.14.147.254@KRBTEST.COM
I20260929 22:52:45.139647 14927 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhWOCLk/build/release/bin/kudu as pid 28400
I20260929 22:52:45.139737 14927 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790722213368738-14927-0/minicluster-data/master-2/wal/instance
I20260929 22:52:45.139930 14927 external_mini_cluster.cc:1428] Setting key 9b644643956adb40fb750b4b5dc5ceb8
I20260929 22:52:45.143045 28321 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:52:45.134061 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43750 (local address 127.14.147.254:33093)
0929 22:52:45.134188 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0929 22:52:45.134191 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 22:52:45.134517 (+ 326us) server_negotiation.cc:407] Connection header received
0929 22:52:45.135238 (+ 721us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 22:52:45.135239 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 22:52:45.135251 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 22:52:45.135333 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0929 22:52:45.136107 (+ 774us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:52:45.136601 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:52:45.137705 (+ 1104us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:52:45.137925 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:52:45.137995 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 22:52:45.140255 (+ 2260us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 22:52:45.140258 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 22:52:45.140258 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 22:52:45.140267 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0929 22:52:45.141745 (+ 1478us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 22:52:45.142135 (+ 390us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 22:52:45.142137 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 22:52:45.142137 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 22:52:45.142198 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 22:52:45.142428 (+ 230us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 22:52:45.142429 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 22:52:45.142429 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 22:52:45.142567 (+ 138us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 22:52:45.142630 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0929 22:52:45.142859 (+ 229us) server_negotiation.cc:299] Negotiation successful
0929 22:52:45.142943 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260929 22:52:45.143198 28470 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:52:45.134132 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.147.254:33093 (local address 127.0.0.1:43750)
0929 22:52:45.134342 (+ 210us) negotiation.cc:107] Waiting for socket to connect
0929 22:52:45.134399 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0929 22:52:45.135123 (+ 724us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 22:52:45.135514 (+ 391us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 22:52:45.135519 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 22:52:45.135801 (+ 282us) client_negotiation.cc:189] Negotiated authn=SASL
0929 22:52:45.136016 (+ 215us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 22:52:45.136021 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 22:52:45.136782 (+ 761us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 22:52:45.136783 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 22:52:45.137620 (+ 837us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 22:52:45.137623 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 22:52:45.137999 (+ 376us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 22:52:45.138000 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 22:52:45.138111 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 22:52:45.138610 (+ 499us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 22:52:45.138619 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0929 22:52:45.140156 (+ 1537us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 22:52:45.141822 (+ 1666us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 22:52:45.141830 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 22:52:45.141832 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 22:52:45.142064 (+ 232us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 22:52:45.142256 (+ 192us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 22:52:45.142257 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 22:52:45.142257 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 22:52:45.142367 (+ 110us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 22:52:45.142627 (+ 260us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 22:52:45.142630 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 22:52:45.142750 (+ 120us) client_negotiation.cc:769] Sending connection context
0929 22:52:45.142825 (+ 75us) client_negotiation.cc:240] Negotiation successful
0929 22:52:45.142959 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":158,"thread_start_us":91,"threads_started":1}
I20260929 22:52:45.143769 28468 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.14.147.253" port: 40371 } has no permanent_uuid. Determining permanent_uuid...
Sep 29 22:52:45 dist-test-slave-98mk krb5kdc[28237](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790722364, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.147.252@KRBTEST.COM
Sep 29 22:52:45 dist-test-slave-98mk krb5kdc[28237](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790722365, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.147.252@KRBTEST.COM for kudu/127.14.147.253@KRBTEST.COM
I20260929 22:52:45.148597 28472 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:52:45.140315 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52788 (local address 127.14.147.252:33573)
0929 22:52:45.140637 (+ 322us) server_negotiation.cc:206] Beginning negotiation
0929 22:52:45.140642 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 22:52:45.140669 (+ 27us) server_negotiation.cc:407] Connection header received
0929 22:52:45.140692 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 22:52:45.140695 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 22:52:45.140711 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 22:52:45.140792 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0929 22:52:45.141445 (+ 653us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:52:45.142131 (+ 686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:52:45.142994 (+ 863us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:52:45.143454 (+ 460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:52:45.143518 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 22:52:45.145801 (+ 2283us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 22:52:45.145810 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 22:52:45.145811 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 22:52:45.145819 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0929 22:52:45.147484 (+ 1665us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 22:52:45.147776 (+ 292us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 22:52:45.147779 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 22:52:45.147780 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 22:52:45.147834 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 22:52:45.148025 (+ 191us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 22:52:45.148026 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 22:52:45.148026 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 22:52:45.148188 (+ 162us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 22:52:45.148320 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0929 22:52:45.148406 (+ 86us) server_negotiation.cc:299] Negotiation successful
0929 22:52:45.148483 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":286,"thread_start_us":85,"threads_started":1}
I20260929 22:52:45.150110 28470 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:52:45.144039 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.147.253:40371 (local address 127.0.0.1:56102)
0929 22:52:45.144094 (+ 55us) negotiation.cc:107] Waiting for socket to connect
0929 22:52:45.144106 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0929 22:52:45.144146 (+ 40us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 22:52:45.144265 (+ 119us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 22:52:45.144265 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 22:52:45.144408 (+ 143us) client_negotiation.cc:189] Negotiated authn=SASL
0929 22:52:45.144468 (+ 60us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 22:52:45.144470 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 22:52:45.145048 (+ 578us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 22:52:45.145050 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 22:52:45.145592 (+ 542us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 22:52:45.145595 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 22:52:45.145926 (+ 331us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 22:52:45.145927 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 22:52:45.145993 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 22:52:45.146374 (+ 381us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 22:52:45.146377 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0929 22:52:45.147590 (+ 1213us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 22:52:45.149083 (+ 1493us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 22:52:45.149084 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 22:52:45.149084 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 22:52:45.149260 (+ 176us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 22:52:45.149509 (+ 249us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 22:52:45.149510 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 22:52:45.149511 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 22:52:45.149574 (+ 63us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 22:52:45.149840 (+ 266us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 22:52:45.149841 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 22:52:45.149881 (+ 40us) client_negotiation.cc:769] Sending connection context
0929 22:52:45.149930 (+ 49us) client_negotiation.cc:240] Negotiation successful
0929 22:52:45.150019 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":35}
I20260929 22:52:45.150136 28397 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:52:45.144089 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56102 (local address 127.14.147.253:40371)
0929 22:52:45.144131 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0929 22:52:45.144133 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 22:52:45.144141 (+ 8us) server_negotiation.cc:407] Connection header received
0929 22:52:45.144199 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 22:52:45.144200 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 22:52:45.144208 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 22:52:45.144249 (+ 41us) server_negotiation.cc:226] Negotiated authn=SASL
0929 22:52:45.144529 (+ 280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:52:45.144978 (+ 449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:52:45.145663 (+ 685us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:52:45.145871 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:52:45.145924 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 22:52:45.147675 (+ 1751us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 22:52:45.147676 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 22:52:45.147677 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 22:52:45.147683 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0929 22:52:45.149014 (+ 1331us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 22:52:45.149361 (+ 347us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 22:52:45.149362 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 22:52:45.149363 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 22:52:45.149425 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 22:52:45.149644 (+ 219us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 22:52:45.149645 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 22:52:45.149645 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 22:52:45.149768 (+ 123us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 22:52:45.149804 (+ 36us) server_negotiation.cc:1035] Waiting for connection context
0929 22:52:45.149984 (+ 180us) server_negotiation.cc:299] Negotiation successful
0929 22:52:45.150057 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260929 22:52:45.150724 28468 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.14.147.252" port: 33573 } has no permanent_uuid. Determining permanent_uuid...
Sep 29 22:52:45 dist-test-slave-98mk krb5kdc[28237](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790722364, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.147.254@KRBTEST.COM
Sep 29 22:52:45 dist-test-slave-98mk krb5kdc[28237](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790722365, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.147.252@KRBTEST.COM for kudu/127.14.147.252@KRBTEST.COM
I20260929 22:52:45.156764 28321 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:52:45.149674 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43758 (local address 127.14.147.254:33093)
0929 22:52:45.149741 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0929 22:52:45.149743 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 22:52:45.149751 (+ 8us) server_negotiation.cc:407] Connection header received
0929 22:52:45.149774 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 22:52:45.149775 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 22:52:45.149785 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 22:52:45.149845 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0929 22:52:45.150301 (+ 456us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:52:45.150903 (+ 602us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:52:45.151627 (+ 724us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:52:45.151929 (+ 302us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:52:45.151991 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 22:52:45.154187 (+ 2196us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 22:52:45.154188 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 22:52:45.154189 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 22:52:45.154197 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0929 22:52:45.155717 (+ 1520us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 22:52:45.156047 (+ 330us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 22:52:45.156048 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 22:52:45.156048 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 22:52:45.156103 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 22:52:45.156268 (+ 165us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 22:52:45.156268 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 22:52:45.156269 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 22:52:45.156385 (+ 116us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 22:52:45.156562 (+ 177us) server_negotiation.cc:1035] Waiting for connection context
0929 22:52:45.156600 (+ 38us) server_negotiation.cc:299] Negotiation successful
0929 22:52:45.156674 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260929 22:52:45.156771 28470 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:52:45.150984 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.147.252:33573 (local address 127.0.0.1:52800)
0929 22:52:45.151030 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0929 22:52:45.151040 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0929 22:52:45.151074 (+ 34us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 22:52:45.151185 (+ 111us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 22:52:45.151186 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 22:52:45.151307 (+ 121us) client_negotiation.cc:189] Negotiated authn=SASL
0929 22:52:45.151333 (+ 26us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 22:52:45.151334 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 22:52:45.152098 (+ 764us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 22:52:45.152099 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 22:52:45.152660 (+ 561us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 22:52:45.152662 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 22:52:45.152996 (+ 334us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 22:52:45.152997 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 22:52:45.153043 (+ 46us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 22:52:45.153361 (+ 318us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 22:52:45.153365 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0929 22:52:45.154583 (+ 1218us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 22:52:45.155829 (+ 1246us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 22:52:45.155830 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 22:52:45.155830 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 22:52:45.155982 (+ 152us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 22:52:45.156145 (+ 163us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 22:52:45.156146 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 22:52:45.156146 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 22:52:45.156203 (+ 57us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 22:52:45.156429 (+ 226us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 22:52:45.156430 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 22:52:45.156473 (+ 43us) client_negotiation.cc:769] Sending connection context
0929 22:52:45.156537 (+ 64us) client_negotiation.cc:240] Negotiation successful
0929 22:52:45.156641 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":29}
I20260929 22:52:45.156831 28472 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:52:45.151076 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52800 (local address 127.14.147.252:33573)
0929 22:52:45.151111 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0929 22:52:45.151112 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0929 22:52:45.151118 (+ 6us) server_negotiation.cc:407] Connection header received
0929 22:52:45.151133 (+ 15us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 22:52:45.151134 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 22:52:45.151142 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 22:52:45.151184 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0929 22:52:45.151396 (+ 212us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:52:45.152046 (+ 650us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:52:45.152731 (+ 685us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:52:45.152939 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:52:45.152991 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 22:52:45.154654 (+ 1663us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 22:52:45.154655 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 22:52:45.154655 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 22:52:45.154661 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0929 22:52:45.155755 (+ 1094us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 22:52:45.156043 (+ 288us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 22:52:45.156044 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 22:52:45.156045 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 22:52:45.156093 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 22:52:45.156257 (+ 164us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 22:52:45.156258 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 22:52:45.156258 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 22:52:45.156369 (+ 111us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 22:52:45.156421 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0929 22:52:45.156567 (+ 146us) server_negotiation.cc:299] Negotiation successful
0929 22:52:45.156696 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":54,"server-negotiator.queue_time_us":22}
I20260929 22:52:45.158478 28468 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2d10da5a5bf3463694927f4485c8d83e: Bootstrap starting.
I20260929 22:52:45.159508 28468 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2d10da5a5bf3463694927f4485c8d83e: Neither blocks nor log segments found. Creating new log.
I20260929 22:52:45.159801 28468 log.cc:824] T 00000000000000000000000000000000 P 2d10da5a5bf3463694927f4485c8d83e: Log is configured to *not* fsync() on all Append() calls
I20260929 22:52:45.160579 28468 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2d10da5a5bf3463694927f4485c8d83e: No bootstrap required, opened a new log
Sep 29 22:52:45 dist-test-slave-98mk krb5kdc[28237](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790722364, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.147.253@KRBTEST.COM
I20260929 22:52:45.162357 28468 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2d10da5a5bf3463694927f4485c8d83e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b35099a272c43798770ed452b8c6eab" member_type: VOTER last_known_addr { host: "127.14.147.254" port: 33093 } } peers { permanent_uuid: "303754d07d40449c97a83aede259ba4d" member_type: VOTER last_known_addr { host: "127.14.147.253" port: 40371 } } peers { permanent_uuid: "2d10da5a5bf3463694927f4485c8d83e" member_type: VOTER last_known_addr { host: "127.14.147.252" port: 33573 } }
I20260929 22:52:45.162495 28468 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2d10da5a5bf3463694927f4485c8d83e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 22:52:45.162529 28468 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2d10da5a5bf3463694927f4485c8d83e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2d10da5a5bf3463694927f4485c8d83e, State: Initialized, Role: FOLLOWER
I20260929 22:52:45.162662 28468 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2d10da5a5bf3463694927f4485c8d83e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b35099a272c43798770ed452b8c6eab" member_type: VOTER last_known_addr { host: "127.14.147.254" port: 33093 } } peers { permanent_uuid: "303754d07d40449c97a83aede259ba4d" member_type: VOTER last_known_addr { host: "127.14.147.253" port: 40371 } } peers { permanent_uuid: "2d10da5a5bf3463694927f4485c8d83e" member_type: VOTER last_known_addr { host: "127.14.147.252" port: 33573 } }
I20260929 22:52:45.162957 28474 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2d10da5a5bf3463694927f4485c8d83e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b35099a272c43798770ed452b8c6eab" member_type: VOTER last_known_addr { host: "127.14.147.254" port: 33093 } } peers { permanent_uuid: "303754d07d40449c97a83aede259ba4d" member_type: VOTER last_known_addr { host: "127.14.147.253" port: 40371 } } peers { permanent_uuid: "2d10da5a5bf3463694927f4485c8d83e" member_type: VOTER last_known_addr { host: "127.14.147.252" port: 33573 } } }
I20260929 22:52:45.163003 28468 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2d10da5a5bf3463694927f4485c8d83e [sys.catalog]: configured and running, proceeding with master startup.
I20260929 22:52:45.163050 28474 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2d10da5a5bf3463694927f4485c8d83e [sys.catalog]: This master's current role is: FOLLOWER
W20260929 22:52:45.164853 28485 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 2d10da5a5bf3463694927f4485c8d83e: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 22:52:45.164969 28485 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260929 22:52:45.165093 28397 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:52:45.157466 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56108 (local address 127.14.147.253:40371)
0929 22:52:45.157521 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0929 22:52:45.157522 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0929 22:52:45.157553 (+ 31us) server_negotiation.cc:407] Connection header received
0929 22:52:45.157626 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 22:52:45.157627 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 22:52:45.157635 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 22:52:45.157678 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0929 22:52:45.158246 (+ 568us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:52:45.158758 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:52:45.159671 (+ 913us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:52:45.159849 (+ 178us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:52:45.159910 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 22:52:45.162298 (+ 2388us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 22:52:45.162300 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 22:52:45.162301 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 22:52:45.162309 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0929 22:52:45.163786 (+ 1477us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 22:52:45.164134 (+ 348us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 22:52:45.164135 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 22:52:45.164135 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 22:52:45.164204 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 22:52:45.164413 (+ 209us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 22:52:45.164415 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 22:52:45.164415 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 22:52:45.164548 (+ 133us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 22:52:45.164717 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0929 22:52:45.164917 (+ 200us) server_negotiation.cc:299] Negotiation successful
0929 22:52:45.165006 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260929 22:52:45.166582 14927 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 22:52:45.165107 28400 master_runner.cc:186] Error getting master registration for 127.14.147.254:33093: OK, instance_id { permanent_uuid: "1b35099a272c43798770ed452b8c6eab" instance_seqno: 1790722364834122 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260929 22:52:45.167697 28400 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260929 22:52:45.176110 28397 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:52:45.168309 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56112 (local address 127.14.147.253:40371)
0929 22:52:45.168377 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0929 22:52:45.168379 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 22:52:45.168597 (+ 218us) server_negotiation.cc:407] Connection header received
0929 22:52:45.168676 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 22:52:45.168677 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 22:52:45.168711 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 22:52:45.168770 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0929 22:52:45.169933 (+ 1163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:52:45.170879 (+ 946us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:52:45.171803 (+ 924us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:52:45.172088 (+ 285us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:52:45.172130 (+ 42us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 22:52:45.173533 (+ 1403us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 22:52:45.173534 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 22:52:45.173535 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 22:52:45.173541 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0929 22:52:45.174986 (+ 1445us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 22:52:45.175345 (+ 359us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 22:52:45.175346 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 22:52:45.175347 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 22:52:45.175403 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 22:52:45.175598 (+ 195us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 22:52:45.175598 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 22:52:45.175599 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 22:52:45.175732 (+ 133us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 22:52:45.175834 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0929 22:52:45.175940 (+ 106us) server_negotiation.cc:299] Negotiation successful
0929 22:52:45.176013 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260929 22:52:45.176402 28321 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:52:45.168127 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43766 (local address 127.14.147.254:33093)
0929 22:52:45.168172 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0929 22:52:45.168174 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 22:52:45.168259 (+ 85us) server_negotiation.cc:407] Connection header received
0929 22:52:45.168357 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 22:52:45.168358 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 22:52:45.168369 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 22:52:45.168399 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0929 22:52:45.169164 (+ 765us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:52:45.169871 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:52:45.170926 (+ 1055us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:52:45.171234 (+ 308us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:52:45.171300 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 22:52:45.173203 (+ 1903us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 22:52:45.173204 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 22:52:45.173205 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 22:52:45.173213 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0929 22:52:45.175144 (+ 1931us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 22:52:45.175548 (+ 404us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 22:52:45.175550 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 22:52:45.175550 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 22:52:45.175611 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 22:52:45.175807 (+ 196us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 22:52:45.175808 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 22:52:45.175808 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 22:52:45.175954 (+ 146us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 22:52:45.176022 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0929 22:52:45.176227 (+ 205us) server_negotiation.cc:299] Negotiation successful
0929 22:52:45.176309 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260929 22:52:45.177314 28318 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.14.147.252" port: 33573 } attempt: 2
Sep 29 22:52:45 dist-test-slave-98mk krb5kdc[28237](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790722364, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.147.254@KRBTEST.COM for kudu/127.14.147.252@KRBTEST.COM
I20260929 22:52:45.182765 28472 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:52:45.168474 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52806 (local address 127.14.147.252:33573)
0929 22:52:45.168883 (+ 409us) server_negotiation.cc:206] Beginning negotiation
0929 22:52:45.168885 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 22:52:45.176175 (+ 7290us) server_negotiation.cc:407] Connection header received
0929 22:52:45.176605 (+ 430us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 22:52:45.176607 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 22:52:45.176618 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 22:52:45.176644 (+ 26us) server_negotiation.cc:226] Negotiated authn=SASL
0929 22:52:45.177173 (+ 529us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:52:45.177855 (+ 682us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:52:45.178624 (+ 769us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:52:45.178822 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:52:45.178878 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 22:52:45.180054 (+ 1176us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 22:52:45.180055 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 22:52:45.180056 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 22:52:45.180063 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0929 22:52:45.181606 (+ 1543us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 22:52:45.181991 (+ 385us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 22:52:45.181992 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 22:52:45.181992 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 22:52:45.182067 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 22:52:45.182242 (+ 175us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 22:52:45.182243 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 22:52:45.182243 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 22:52:45.182386 (+ 143us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 22:52:45.182473 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0929 22:52:45.182596 (+ 123us) server_negotiation.cc:299] Negotiation successful
0929 22:52:45.182673 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":363}
I20260929 22:52:45.185595 28494 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:52:45.177692 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52822 (local address 127.14.147.252:33573)
0929 22:52:45.177918 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0929 22:52:45.177920 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 22:52:45.177927 (+ 7us) server_negotiation.cc:407] Connection header received
0929 22:52:45.177955 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 22:52:45.177955 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 22:52:45.177965 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 22:52:45.178038 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0929 22:52:45.178324 (+ 286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:52:45.179141 (+ 817us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:52:45.180190 (+ 1049us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:52:45.181051 (+ 861us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:52:45.181369 (+ 318us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 22:52:45.183326 (+ 1957us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 22:52:45.183327 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 22:52:45.183327 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 22:52:45.183332 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0929 22:52:45.184640 (+ 1308us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 22:52:45.184990 (+ 350us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 22:52:45.184990 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 22:52:45.184991 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 22:52:45.185035 (+ 44us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 22:52:45.185205 (+ 170us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 22:52:45.185206 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 22:52:45.185206 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 22:52:45.185291 (+ 85us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 22:52:45.185331 (+ 40us) server_negotiation.cc:1035] Waiting for connection context
0929 22:52:45.185465 (+ 134us) server_negotiation.cc:299] Negotiation successful
0929 22:52:45.185518 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":158,"thread_start_us":95,"threads_started":1}
I20260929 22:52:45.185643 28320 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:52:45.177575 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.147.252:33573 (local address 127.0.0.1:52822)
0929 22:52:45.177673 (+ 98us) negotiation.cc:107] Waiting for socket to connect
0929 22:52:45.177683 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0929 22:52:45.177724 (+ 41us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 22:52:45.178054 (+ 330us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 22:52:45.178055 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 22:52:45.178208 (+ 153us) client_negotiation.cc:189] Negotiated authn=SASL
0929 22:52:45.178249 (+ 41us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 22:52:45.178251 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 22:52:45.179237 (+ 986us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 22:52:45.179238 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 22:52:45.180075 (+ 837us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 22:52:45.180078 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 22:52:45.181145 (+ 1067us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 22:52:45.181146 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 22:52:45.181207 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 22:52:45.181586 (+ 379us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 22:52:45.181591 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0929 22:52:45.183257 (+ 1666us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 22:52:45.184711 (+ 1454us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 22:52:45.184712 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 22:52:45.184713 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 22:52:45.184925 (+ 212us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 22:52:45.185094 (+ 169us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 22:52:45.185095 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 22:52:45.185095 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 22:52:45.185155 (+ 60us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 22:52:45.185335 (+ 180us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 22:52:45.185336 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 22:52:45.185381 (+ 45us) client_negotiation.cc:769] Sending connection context
0929 22:52:45.185448 (+ 67us) client_negotiation.cc:240] Negotiation successful
0929 22:52:45.185542 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":78}
I20260929 22:52:45.186956 28318 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1b35099a272c43798770ed452b8c6eab: Bootstrap starting.
I20260929 22:52:45.187652 28318 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1b35099a272c43798770ed452b8c6eab: Neither blocks nor log segments found. Creating new log.
I20260929 22:52:45.187896 28318 log.cc:824] T 00000000000000000000000000000000 P 1b35099a272c43798770ed452b8c6eab: Log is configured to *not* fsync() on all Append() calls
I20260929 22:52:45.188666 28318 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1b35099a272c43798770ed452b8c6eab: No bootstrap required, opened a new log
I20260929 22:52:45.190093 28318 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1b35099a272c43798770ed452b8c6eab [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b35099a272c43798770ed452b8c6eab" member_type: VOTER last_known_addr { host: "127.14.147.254" port: 33093 } } peers { permanent_uuid: "303754d07d40449c97a83aede259ba4d" member_type: VOTER last_known_addr { host: "127.14.147.253" port: 40371 } } peers { permanent_uuid: "2d10da5a5bf3463694927f4485c8d83e" member_type: VOTER last_known_addr { host: "127.14.147.252" port: 33573 } }
I20260929 22:52:45.190200 28318 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1b35099a272c43798770ed452b8c6eab [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 22:52:45.190219 28318 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1b35099a272c43798770ed452b8c6eab [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1b35099a272c43798770ed452b8c6eab, State: Initialized, Role: FOLLOWER
I20260929 22:52:45.190321 28318 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1b35099a272c43798770ed452b8c6eab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b35099a272c43798770ed452b8c6eab" member_type: VOTER last_known_addr { host: "127.14.147.254" port: 33093 } } peers { permanent_uuid: "303754d07d40449c97a83aede259ba4d" member_type: VOTER last_known_addr { host: "127.14.147.253" port: 40371 } } peers { permanent_uuid: "2d10da5a5bf3463694927f4485c8d83e" member_type: VOTER last_known_addr { host: "127.14.147.252" port: 33573 } }
I20260929 22:52:45.190603 28318 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1b35099a272c43798770ed452b8c6eab [sys.catalog]: configured and running, proceeding with master startup.
I20260929 22:52:45.190604 28496 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1b35099a272c43798770ed452b8c6eab [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b35099a272c43798770ed452b8c6eab" member_type: VOTER last_known_addr { host: "127.14.147.254" port: 33093 } } peers { permanent_uuid: "303754d07d40449c97a83aede259ba4d" member_type: VOTER last_known_addr { host: "127.14.147.253" port: 40371 } } peers { permanent_uuid: "2d10da5a5bf3463694927f4485c8d83e" member_type: VOTER last_known_addr { host: "127.14.147.252" port: 33573 } } }
I20260929 22:52:45.190738 28496 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1b35099a272c43798770ed452b8c6eab [sys.catalog]: This master's current role is: FOLLOWER
W20260929 22:52:45.192175 28507 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 1b35099a272c43798770ed452b8c6eab: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 22:52:45.192258 28507 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260929 22:52:45.192780 28250 master_runner.cc:186] Error getting master registration for 127.14.147.253:40371: OK, instance_id { permanent_uuid: "303754d07d40449c97a83aede259ba4d" instance_seqno: 1790722364977267 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260929 22:52:45.195073 28250 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260929 22:52:45.211836 28393 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.14.147.252" port: 33573 } attempt: 3
Sep 29 22:52:45 dist-test-slave-98mk krb5kdc[28237](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790722364, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.147.253@KRBTEST.COM for kudu/127.14.147.252@KRBTEST.COM
I20260929 22:52:45.220750 28395 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:52:45.212169 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.147.252:33573 (local address 127.0.0.1:52836)
0929 22:52:45.212230 (+ 61us) negotiation.cc:107] Waiting for socket to connect
0929 22:52:45.212241 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0929 22:52:45.212279 (+ 38us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 22:52:45.212415 (+ 136us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 22:52:45.212416 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 22:52:45.212608 (+ 192us) client_negotiation.cc:189] Negotiated authn=SASL
0929 22:52:45.212663 (+ 55us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 22:52:45.212665 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 22:52:45.213620 (+ 955us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 22:52:45.213621 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 22:52:45.214430 (+ 809us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 22:52:45.214433 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 22:52:45.214953 (+ 520us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 22:52:45.214955 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 22:52:45.215016 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 22:52:45.215558 (+ 542us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 22:52:45.215564 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0929 22:52:45.217113 (+ 1549us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 22:52:45.219113 (+ 2000us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 22:52:45.219114 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 22:52:45.219115 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 22:52:45.219379 (+ 264us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 22:52:45.219669 (+ 290us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 22:52:45.219669 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 22:52:45.219670 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 22:52:45.219728 (+ 58us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 22:52:45.220350 (+ 622us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 22:52:45.220351 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 22:52:45.220414 (+ 63us) client_negotiation.cc:769] Sending connection context
0929 22:52:45.220494 (+ 80us) client_negotiation.cc:240] Negotiation successful
0929 22:52:45.220639 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":39}
I20260929 22:52:45.221113 28494 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:52:45.212234 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52836 (local address 127.14.147.252:33573)
0929 22:52:45.212280 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0929 22:52:45.212281 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0929 22:52:45.212292 (+ 11us) server_negotiation.cc:407] Connection header received
0929 22:52:45.212325 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 22:52:45.212325 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 22:52:45.212336 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 22:52:45.212404 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0929 22:52:45.212743 (+ 339us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:52:45.213529 (+ 786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:52:45.214536 (+ 1007us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:52:45.214867 (+ 331us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:52:45.214949 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 22:52:45.217214 (+ 2265us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 22:52:45.217215 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 22:52:45.217215 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 22:52:45.217224 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0929 22:52:45.219010 (+ 1786us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 22:52:45.219516 (+ 506us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 22:52:45.219518 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 22:52:45.219518 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 22:52:45.219580 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 22:52:45.219796 (+ 216us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 22:52:45.219797 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 22:52:45.219798 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 22:52:45.220215 (+ 417us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 22:52:45.220368 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0929 22:52:45.220919 (+ 551us) server_negotiation.cc:299] Negotiation successful
0929 22:52:45.221024 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260929 22:52:45.222929 28393 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 303754d07d40449c97a83aede259ba4d: Bootstrap starting.
I20260929 22:52:45.223760 28393 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 303754d07d40449c97a83aede259ba4d: Neither blocks nor log segments found. Creating new log.
I20260929 22:52:45.224076 28393 log.cc:824] T 00000000000000000000000000000000 P 303754d07d40449c97a83aede259ba4d: Log is configured to *not* fsync() on all Append() calls
I20260929 22:52:45.225010 28393 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 303754d07d40449c97a83aede259ba4d: No bootstrap required, opened a new log
I20260929 22:52:45.227949 28393 raft_consensus.cc:348] T 00000000000000000000000000000000 P 303754d07d40449c97a83aede259ba4d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b35099a272c43798770ed452b8c6eab" member_type: VOTER last_known_addr { host: "127.14.147.254" port: 33093 } } peers { permanent_uuid: "303754d07d40449c97a83aede259ba4d" member_type: VOTER last_known_addr { host: "127.14.147.253" port: 40371 } } peers { permanent_uuid: "2d10da5a5bf3463694927f4485c8d83e" member_type: VOTER last_known_addr { host: "127.14.147.252" port: 33573 } }
I20260929 22:52:45.228125 28393 raft_consensus.cc:374] T 00000000000000000000000000000000 P 303754d07d40449c97a83aede259ba4d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 22:52:45.228158 28393 raft_consensus.cc:729] T 00000000000000000000000000000000 P 303754d07d40449c97a83aede259ba4d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 303754d07d40449c97a83aede259ba4d, State: Initialized, Role: FOLLOWER
I20260929 22:52:45.228349 28393 consensus_queue.cc:260] T 00000000000000000000000000000000 P 303754d07d40449c97a83aede259ba4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b35099a272c43798770ed452b8c6eab" member_type: VOTER last_known_addr { host: "127.14.147.254" port: 33093 } } peers { permanent_uuid: "303754d07d40449c97a83aede259ba4d" member_type: VOTER last_known_addr { host: "127.14.147.253" port: 40371 } } peers { permanent_uuid: "2d10da5a5bf3463694927f4485c8d83e" member_type: VOTER last_known_addr { host: "127.14.147.252" port: 33573 } }
I20260929 22:52:45.228713 28393 sys_catalog.cc:564] T 00000000000000000000000000000000 P 303754d07d40449c97a83aede259ba4d [sys.catalog]: configured and running, proceeding with master startup.
I20260929 22:52:45.228664 28510 sys_catalog.cc:455] T 00000000000000000000000000000000 P 303754d07d40449c97a83aede259ba4d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b35099a272c43798770ed452b8c6eab" member_type: VOTER last_known_addr { host: "127.14.147.254" port: 33093 } } peers { permanent_uuid: "303754d07d40449c97a83aede259ba4d" member_type: VOTER last_known_addr { host: "127.14.147.253" port: 40371 } } peers { permanent_uuid: "2d10da5a5bf3463694927f4485c8d83e" member_type: VOTER last_known_addr { host: "127.14.147.252" port: 33573 } } }
I20260929 22:52:45.228889 28510 sys_catalog.cc:458] T 00000000000000000000000000000000 P 303754d07d40449c97a83aede259ba4d [sys.catalog]: This master's current role is: FOLLOWER
W20260929 22:52:45.230644 28521 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 303754d07d40449c97a83aede259ba4d: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 22:52:45.230796 28521 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260929 22:52:45.233210 28325 master_runner.cc:263] No leader master found from master 303754d07d40449c97a83aede259ba4d
I20260929 22:52:45.233279 28325 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260929 22:52:45.323774 28474 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2d10da5a5bf3463694927f4485c8d83e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 22:52:45.323946 28474 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2d10da5a5bf3463694927f4485c8d83e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b35099a272c43798770ed452b8c6eab" member_type: VOTER last_known_addr { host: "127.14.147.254" port: 33093 } } peers { permanent_uuid: "303754d07d40449c97a83aede259ba4d" member_type: VOTER last_known_addr { host: "127.14.147.253" port: 40371 } } peers { permanent_uuid: "2d10da5a5bf3463694927f4485c8d83e" member_type: VOTER last_known_addr { host: "127.14.147.252" port: 33573 } }
I20260929 22:52:45.324235 28474 leader_election.cc:288] T 00000000000000000000000000000000 P 2d10da5a5bf3463694927f4485c8d83e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1b35099a272c43798770ed452b8c6eab (127.14.147.254:33093), 303754d07d40449c97a83aede259ba4d (127.14.147.253:40371)
I20260929 22:52:45.324646 28367 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "2d10da5a5bf3463694927f4485c8d83e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "303754d07d40449c97a83aede259ba4d" is_pre_election: true
I20260929 22:52:45.324832 28367 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 303754d07d40449c97a83aede259ba4d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2d10da5a5bf3463694927f4485c8d83e in term 0.
I20260929 22:52:45.324702 28292 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "2d10da5a5bf3463694927f4485c8d83e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1b35099a272c43798770ed452b8c6eab" is_pre_election: true
I20260929 22:52:45.324961 28292 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 1b35099a272c43798770ed452b8c6eab [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2d10da5a5bf3463694927f4485c8d83e in term 0.
I20260929 22:52:45.325223 28416 leader_election.cc:302] T 00000000000000000000000000000000 P 2d10da5a5bf3463694927f4485c8d83e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2d10da5a5bf3463694927f4485c8d83e, 303754d07d40449c97a83aede259ba4d; no voters:
I20260929 22:52:45.325486 28474 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2d10da5a5bf3463694927f4485c8d83e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 22:52:45.325541 28474 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2d10da5a5bf3463694927f4485c8d83e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 22:52:45.325559 28474 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2d10da5a5bf3463694927f4485c8d83e [term 0 FOLLOWER]: Advancing to term 1
I20260929 22:52:45.326493 28474 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2d10da5a5bf3463694927f4485c8d83e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b35099a272c43798770ed452b8c6eab" member_type: VOTER last_known_addr { host: "127.14.147.254" port: 33093 } } peers { permanent_uuid: "303754d07d40449c97a83aede259ba4d" member_type: VOTER last_known_addr { host: "127.14.147.253" port: 40371 } } peers { permanent_uuid: "2d10da5a5bf3463694927f4485c8d83e" member_type: VOTER last_known_addr { host: "127.14.147.252" port: 33573 } }
I20260929 22:52:45.326627 28474 leader_election.cc:288] T 00000000000000000000000000000000 P 2d10da5a5bf3463694927f4485c8d83e [CANDIDATE]: Term 1 election: Requested vote from peers 1b35099a272c43798770ed452b8c6eab (127.14.147.254:33093), 303754d07d40449c97a83aede259ba4d (127.14.147.253:40371)
I20260929 22:52:45.326851 28367 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "2d10da5a5bf3463694927f4485c8d83e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "303754d07d40449c97a83aede259ba4d"
I20260929 22:52:45.326833 28292 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "2d10da5a5bf3463694927f4485c8d83e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1b35099a272c43798770ed452b8c6eab"
I20260929 22:52:45.326923 28367 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 303754d07d40449c97a83aede259ba4d [term 0 FOLLOWER]: Advancing to term 1
I20260929 22:52:45.326928 28292 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1b35099a272c43798770ed452b8c6eab [term 0 FOLLOWER]: Advancing to term 1
I20260929 22:52:45.327803 28292 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 1b35099a272c43798770ed452b8c6eab [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2d10da5a5bf3463694927f4485c8d83e in term 1.
I20260929 22:52:45.327803 28367 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 303754d07d40449c97a83aede259ba4d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2d10da5a5bf3463694927f4485c8d83e in term 1.
I20260929 22:52:45.327982 28417 leader_election.cc:302] T 00000000000000000000000000000000 P 2d10da5a5bf3463694927f4485c8d83e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1b35099a272c43798770ed452b8c6eab, 2d10da5a5bf3463694927f4485c8d83e; no voters:
I20260929 22:52:45.328266 28474 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2d10da5a5bf3463694927f4485c8d83e [term 1 FOLLOWER]: Leader election won for term 1
I20260929 22:52:45.328433 28474 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2d10da5a5bf3463694927f4485c8d83e [term 1 LEADER]: Becoming Leader. State: Replica: 2d10da5a5bf3463694927f4485c8d83e, State: Running, Role: LEADER
I20260929 22:52:45.328536 28474 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2d10da5a5bf3463694927f4485c8d83e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b35099a272c43798770ed452b8c6eab" member_type: VOTER last_known_addr { host: "127.14.147.254" port: 33093 } } peers { permanent_uuid: "303754d07d40449c97a83aede259ba4d" member_type: VOTER last_known_addr { host: "127.14.147.253" port: 40371 } } peers { permanent_uuid: "2d10da5a5bf3463694927f4485c8d83e" member_type: VOTER last_known_addr { host: "127.14.147.252" port: 33573 } }
I20260929 22:52:45.328946 28474 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2d10da5a5bf3463694927f4485c8d83e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2d10da5a5bf3463694927f4485c8d83e. Latest consensus state: current_term: 1 leader_uuid: "2d10da5a5bf3463694927f4485c8d83e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b35099a272c43798770ed452b8c6eab" member_type: VOTER last_known_addr { host: "127.14.147.254" port: 33093 } } peers { permanent_uuid: "303754d07d40449c97a83aede259ba4d" member_type: VOTER last_known_addr { host: "127.14.147.253" port: 40371 } } peers { permanent_uuid: "2d10da5a5bf3463694927f4485c8d83e" member_type: VOTER last_known_addr { host: "127.14.147.252" port: 33573 } } }
I20260929 22:52:45.329020 28474 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2d10da5a5bf3463694927f4485c8d83e [sys.catalog]: This master's current role is: LEADER
I20260929 22:52:45.329324 28525 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 22:52:45.329528 28525 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 22:52:45.330983 28292 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 1b35099a272c43798770ed452b8c6eab [term 1 FOLLOWER]: Refusing update from remote peer 2d10da5a5bf3463694927f4485c8d83e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 22:52:45.330983 28367 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 303754d07d40449c97a83aede259ba4d [term 1 FOLLOWER]: Refusing update from remote peer 2d10da5a5bf3463694927f4485c8d83e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 22:52:45.331349 28523 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 2d10da5a5bf3463694927f4485c8d83e [LEADER]: Connected to new peer: Peer: permanent_uuid: "303754d07d40449c97a83aede259ba4d" member_type: VOTER last_known_addr { host: "127.14.147.253" port: 40371 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 22:52:45.331502 28474 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 2d10da5a5bf3463694927f4485c8d83e [LEADER]: Connected to new peer: Peer: permanent_uuid: "1b35099a272c43798770ed452b8c6eab" member_type: VOTER last_known_addr { host: "127.14.147.254" port: 33093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260929 22:52:45.332250 28271 tablet.cc:2369] T 00000000000000000000000000000000 P 1b35099a272c43798770ed452b8c6eab: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 22:52:45.332762 28510 sys_catalog.cc:455] T 00000000000000000000000000000000 P 303754d07d40449c97a83aede259ba4d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2d10da5a5bf3463694927f4485c8d83e. Latest consensus state: current_term: 1 leader_uuid: "2d10da5a5bf3463694927f4485c8d83e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b35099a272c43798770ed452b8c6eab" member_type: VOTER last_known_addr { host: "127.14.147.254" port: 33093 } } peers { permanent_uuid: "303754d07d40449c97a83aede259ba4d" member_type: VOTER last_known_addr { host: "127.14.147.253" port: 40371 } } peers { permanent_uuid: "2d10da5a5bf3463694927f4485c8d83e" member_type: VOTER last_known_addr { host: "127.14.147.252" port: 33573 } } }
I20260929 22:52:45.332748 28496 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1b35099a272c43798770ed452b8c6eab [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2d10da5a5bf3463694927f4485c8d83e. Latest consensus state: current_term: 1 leader_uuid: "2d10da5a5bf3463694927f4485c8d83e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b35099a272c43798770ed452b8c6eab" member_type: VOTER last_known_addr { host: "127.14.147.254" port: 33093 } } peers { permanent_uuid: "303754d07d40449c97a83aede259ba4d" member_type: VOTER last_known_addr { host: "127.14.147.253" port: 40371 } } peers { permanent_uuid: "2d10da5a5bf3463694927f4485c8d83e" member_type: VOTER last_known_addr { host: "127.14.147.252" port: 33573 } } }
I20260929 22:52:45.332839 28496 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1b35099a272c43798770ed452b8c6eab [sys.catalog]: This master's current role is: FOLLOWER
I20260929 22:52:45.332839 28510 sys_catalog.cc:458] T 00000000000000000000000000000000 P 303754d07d40449c97a83aede259ba4d [sys.catalog]: This master's current role is: FOLLOWER
I20260929 22:52:45.333953 28474 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2d10da5a5bf3463694927f4485c8d83e [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "2d10da5a5bf3463694927f4485c8d83e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b35099a272c43798770ed452b8c6eab" member_type: VOTER last_known_addr { host: "127.14.147.254" port: 33093 } } peers { permanent_uuid: "303754d07d40449c97a83aede259ba4d" member_type: VOTER last_known_addr { host: "127.14.147.253" port: 40371 } } peers { permanent_uuid: "2d10da5a5bf3463694927f4485c8d83e" member_type: VOTER last_known_addr { host: "127.14.147.252" port: 33573 } } }
I20260929 22:52:45.334038 28474 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2d10da5a5bf3463694927f4485c8d83e [sys.catalog]: This master's current role is: LEADER
I20260929 22:52:45.334529 28496 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1b35099a272c43798770ed452b8c6eab [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "2d10da5a5bf3463694927f4485c8d83e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b35099a272c43798770ed452b8c6eab" member_type: VOTER last_known_addr { host: "127.14.147.254" port: 33093 } } peers { permanent_uuid: "303754d07d40449c97a83aede259ba4d" member_type: VOTER last_known_addr { host: "127.14.147.253" port: 40371 } } peers { permanent_uuid: "2d10da5a5bf3463694927f4485c8d83e" member_type: VOTER last_known_addr { host: "127.14.147.252" port: 33573 } } }
I20260929 22:52:45.334609 28496 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1b35099a272c43798770ed452b8c6eab [sys.catalog]: This master's current role is: FOLLOWER
I20260929 22:52:45.334726 28474 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2d10da5a5bf3463694927f4485c8d83e [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "2d10da5a5bf3463694927f4485c8d83e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b35099a272c43798770ed452b8c6eab" member_type: VOTER last_known_addr { host: "127.14.147.254" port: 33093 } } peers { permanent_uuid: "303754d07d40449c97a83aede259ba4d" member_type: VOTER last_known_addr { host: "127.14.147.253" port: 40371 } } peers { permanent_uuid: "2d10da5a5bf3463694927f4485c8d83e" member_type: VOTER last_known_addr { host: "127.14.147.252" port: 33573 } } }
I20260929 22:52:45.334793 28474 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2d10da5a5bf3463694927f4485c8d83e [sys.catalog]: This master's current role is: LEADER
I20260929 22:52:45.335186 28510 sys_catalog.cc:455] T 00000000000000000000000000000000 P 303754d07d40449c97a83aede259ba4d [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "2d10da5a5bf3463694927f4485c8d83e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b35099a272c43798770ed452b8c6eab" member_type: VOTER last_known_addr { host: "127.14.147.254" port: 33093 } } peers { permanent_uuid: "303754d07d40449c97a83aede259ba4d" member_type: VOTER last_known_addr { host: "127.14.147.253" port: 40371 } } peers { permanent_uuid: "2d10da5a5bf3463694927f4485c8d83e" member_type: VOTER last_known_addr { host: "127.14.147.252" port: 33573 } } }
I20260929 22:52:45.335266 28510 sys_catalog.cc:458] T 00000000000000000000000000000000 P 303754d07d40449c97a83aede259ba4d [sys.catalog]: This master's current role is: FOLLOWER
I20260929 22:52:45.335531 28528 mvcc.cc:205] Tried to move back new op lower bound from 7334798808393113600 to 7334798808386449408. Current Snapshot: MvccSnapshot[applied={T|T < 7334798808393113600}]
I20260929 22:52:45.335793 28525 catalog_manager.cc:1347] Generated new cluster ID: a75e621269254c90a8298807757fb8f6
I20260929 22:52:45.335839 28525 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 22:52:45.340901 28525 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 22:52:45.341555 28525 catalog_manager.cc:1504] Loading token signing keys...
I20260929 22:52:45.350064 28525 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2d10da5a5bf3463694927f4485c8d83e: Generated new TSK 0
I20260929 22:52:45.350262 28525 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 22:52:46.192705 28507 catalog_manager.cc:1259] Loaded cluster ID: a75e621269254c90a8298807757fb8f6
I20260929 22:52:46.192783 28507 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 1b35099a272c43798770ed452b8c6eab: loading cluster ID for follower catalog manager: success
I20260929 22:52:46.193539 28507 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 1b35099a272c43798770ed452b8c6eab: acquiring CA information for follower catalog manager: success
I20260929 22:52:46.193801 28507 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 1b35099a272c43798770ed452b8c6eab: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260929 22:52:46.231292 28521 catalog_manager.cc:1259] Loaded cluster ID: a75e621269254c90a8298807757fb8f6
I20260929 22:52:46.231366 28521 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 303754d07d40449c97a83aede259ba4d: loading cluster ID for follower catalog manager: success
I20260929 22:52:46.232169 28521 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 303754d07d40449c97a83aede259ba4d: acquiring CA information for follower catalog manager: success
I20260929 22:52:46.232434 28521 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 303754d07d40449c97a83aede259ba4d: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260929 22:52:46.407080 14927 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhWOCLk/build/release/bin/kudu with pid 28250
I20260929 22:52:46.411791 14927 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhWOCLk/build/release/bin/kudu with pid 28325
I20260929 22:52:46.416035 14927 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhWOCLk/build/release/bin/kudu with pid 28400
2026-09-29T22:52:46Z chronyd exiting
[ OK ] SecurityITest.IPKICACert (2763 ms)
[ RUN ] SecurityITest.IPKICACertWebServerEndPoints
Loading random data
Initializing database '/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790722213368738-14927-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 22:52:46 dist-test-slave-98mk krb5kdc[28543](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 22:52:46 dist-test-slave-98mk krb5kdc[28543](info): set up 2 sockets
Sep 29 22:52:46 dist-test-slave-98mk krb5kdc[28543](info): commencing operation
krb5kdc: starting...
W20260929 22:52:47.363883 14927 mini_kdc.cc:121] Time spent starting KDC: real 0.928s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790722213368738-14927-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790722213368738-14927-0/krb5kdc/test-user.keytab.
Sep 29 22:52:47 dist-test-slave-98mk krb5kdc[28543](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790722367, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T22:52:47Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T22:52:47Z Disabled control of system clock
WARNING: no policy specified for kudu/127.14.147.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.147.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.147.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790722213368738-14927-0/krb5kdc/kudu_127.14.147.254.keytab.
Entry for principal kudu/127.14.147.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790722213368738-14927-0/krb5kdc/kudu_127.14.147.254.keytab.
I20260929 22:52:47.480377 14927 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhWOCLk/build/release/bin/kudu
/tmp/dist-test-taskhWOCLk/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790722213368738-14927-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790722213368738-14927-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790722213368738-14927-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790722213368738-14927-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.147.254:44925
--webserver_interface=127.14.147.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790722213368738-14927-0/krb5kdc/kudu_127.14.147.254.keytab
--principal=kudu/127.14.147.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.147.212:46489
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.147.254:44925
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790722213368738-14927-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790722213368738-14927-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790722213368738-14927-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 22:52:47.564568 28556 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 22:52:47.564801 28556 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 22:52:47.564836 28556 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 22:52:47.566077 28556 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 22:52:47.566123 28556 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 22:52:47.566140 28556 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 22:52:47.566156 28556 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 22:52:47.566172 28556 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 22:52:47.567745 28556 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.147.212:46489
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790722213368738-14927-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790722213368738-14927-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.14.147.254:44925
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790722213368738-14927-0/krb5kdc/kudu_127.14.147.254.keytab
--principal=kudu/127.14.147.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.147.254:44925
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790722213368738-14927-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.14.147.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28556
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790722213368738-14927-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 21:58:54 UTC on ccycloud.quasar-xguoti.root.comops.site
build id 4681390
I20260929 22:52:47.568001 28556 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 22:52:47.568212 28556 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 22:52:47.572985 28561 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 22:52:47.573076 28556 server_base.cc:1034] running on GCE node
W20260929 22:52:47.573000 28564 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 22:52:47.572993 28562 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 22:52:47.573477 28556 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 22:52:47.573760 28556 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 22:52:47.574934 28556 hybrid_clock.cc:648] HybridClock initialized: now 1790722367574920 us; error 29 us; skew 500 ppm
Sep 29 22:52:47 dist-test-slave-98mk krb5kdc[28543](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790722367, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.147.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 22:52:47.577416 28556 init.cc:377] Logged in from keytab as kudu/127.14.147.254@KRBTEST.COM (short username kudu)
I20260929 22:52:47.577872 28556 webserver.cc:460] Webserver started at http://127.14.147.254:40151/ using document root <none> and password file <none>
I20260929 22:52:47.578075 28556 fs_manager.cc:359] Metadata directory not provided
I20260929 22:52:47.578123 28556 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 22:52:47.578249 28556 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 22:52:47.579326 28556 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790722213368738-14927-0/minicluster-data/master-0/data/instance:
uuid: "0ed547e519a342a395a6ef2972e68592"
format_stamp: "Formatted at 2026-09-29 22:52:47 on dist-test-slave-98mk"
server_key: "27bb3f3ee64ceb4a56ae4734f71d8b20"
server_key_iv: "f030376daff88b5302ae5628250dca1b"
server_key_version: "encryptionkey@0"
I20260929 22:52:47.579721 28556 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790722213368738-14927-0/minicluster-data/master-0/wal/instance:
uuid: "0ed547e519a342a395a6ef2972e68592"
format_stamp: "Formatted at 2026-09-29 22:52:47 on dist-test-slave-98mk"
server_key: "27bb3f3ee64ceb4a56ae4734f71d8b20"
server_key_iv: "f030376daff88b5302ae5628250dca1b"
server_key_version: "encryptionkey@0"
I20260929 22:52:47.581286 28556 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260929 22:52:47.582031 28571 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 22:52:47.582204 28556 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260929 22:52:47.582294 28556 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790722213368738-14927-0/minicluster-data/master-0/data,/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790722213368738-14927-0/minicluster-data/master-0/wal
uuid: "0ed547e519a342a395a6ef2972e68592"
format_stamp: "Formatted at 2026-09-29 22:52:47 on dist-test-slave-98mk"
server_key: "27bb3f3ee64ceb4a56ae4734f71d8b20"
server_key_iv: "f030376daff88b5302ae5628250dca1b"
server_key_version: "encryptionkey@0"
I20260929 22:52:47.582371 28556 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790722213368738-14927-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790722213368738-14927-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790722213368738-14927-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 22:52:47.600616 28556 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 22:52:47.602672 28556 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 22:52:47.602828 28556 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 22:52:47.606993 28556 rpc_server.cc:307] RPC server started. Bound to: 127.14.147.254:44925
I20260929 22:52:47.607028 28623 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.147.254:44925 every 8 connection(s)
I20260929 22:52:47.607367 28556 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790722213368738-14927-0/minicluster-data/master-0/data/info.pb
I20260929 22:52:47.608237 28624 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 22:52:47.610543 28624 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0ed547e519a342a395a6ef2972e68592: Bootstrap starting.
I20260929 22:52:47.611140 28624 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0ed547e519a342a395a6ef2972e68592: Neither blocks nor log segments found. Creating new log.
I20260929 22:52:47.611447 28624 log.cc:824] T 00000000000000000000000000000000 P 0ed547e519a342a395a6ef2972e68592: Log is configured to *not* fsync() on all Append() calls
I20260929 22:52:47.612139 28624 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0ed547e519a342a395a6ef2972e68592: No bootstrap required, opened a new log
I20260929 22:52:47.613739 28624 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0ed547e519a342a395a6ef2972e68592 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0ed547e519a342a395a6ef2972e68592" member_type: VOTER last_known_addr { host: "127.14.147.254" port: 44925 } }
I20260929 22:52:47.613896 28624 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0ed547e519a342a395a6ef2972e68592 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 22:52:47.613919 28624 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0ed547e519a342a395a6ef2972e68592 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0ed547e519a342a395a6ef2972e68592, State: Initialized, Role: FOLLOWER
I20260929 22:52:47.614059 28624 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0ed547e519a342a395a6ef2972e68592 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0ed547e519a342a395a6ef2972e68592" member_type: VOTER last_known_addr { host: "127.14.147.254" port: 44925 } }
I20260929 22:52:47.614128 28624 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0ed547e519a342a395a6ef2972e68592 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 22:52:47.614151 28624 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0ed547e519a342a395a6ef2972e68592 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 22:52:47.614200 28624 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0ed547e519a342a395a6ef2972e68592 [term 0 FOLLOWER]: Advancing to term 1
I20260929 22:52:47.614776 28624 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0ed547e519a342a395a6ef2972e68592 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0ed547e519a342a395a6ef2972e68592" member_type: VOTER last_known_addr { host: "127.14.147.254" port: 44925 } }
I20260929 22:52:47.614876 28624 leader_election.cc:302] T 00000000000000000000000000000000 P 0ed547e519a342a395a6ef2972e68592 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0ed547e519a342a395a6ef2972e68592; no voters:
I20260929 22:52:47.615059 28624 leader_election.cc:288] T 00000000000000000000000000000000 P 0ed547e519a342a395a6ef2972e68592 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 22:52:47.615104 28627 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0ed547e519a342a395a6ef2972e68592 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 22:52:47.615248 28627 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0ed547e519a342a395a6ef2972e68592 [term 1 LEADER]: Becoming Leader. State: Replica: 0ed547e519a342a395a6ef2972e68592, State: Running, Role: LEADER
I20260929 22:52:47.615317 28624 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0ed547e519a342a395a6ef2972e68592 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 22:52:47.615340 28627 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0ed547e519a342a395a6ef2972e68592 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0ed547e519a342a395a6ef2972e68592" member_type: VOTER last_known_addr { host: "127.14.147.254" port: 44925 } }
I20260929 22:52:47.615767 14927 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhWOCLk/build/release/bin/kudu as pid 28556
I20260929 22:52:47.615870 14927 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790722213368738-14927-0/minicluster-data/master-0/wal/instance
I20260929 22:52:47.615955 28628 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0ed547e519a342a395a6ef2972e68592 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0ed547e519a342a395a6ef2972e68592" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0ed547e519a342a395a6ef2972e68592" member_type: VOTER last_known_addr { host: "127.14.147.254" port: 44925 } } }
I20260929 22:52:47.616029 28628 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0ed547e519a342a395a6ef2972e68592 [sys.catalog]: This master's current role is: LEADER
I20260929 22:52:47.616050 14927 external_mini_cluster.cc:1428] Setting key 0d911514cc66c1607c846d1edd37a10a
I20260929 22:52:47.616322 28629 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0ed547e519a342a395a6ef2972e68592 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0ed547e519a342a395a6ef2972e68592. Latest consensus state: current_term: 1 leader_uuid: "0ed547e519a342a395a6ef2972e68592" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0ed547e519a342a395a6ef2972e68592" member_type: VOTER last_known_addr { host: "127.14.147.254" port: 44925 } } }
I20260929 22:52:47.616415 28629 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0ed547e519a342a395a6ef2972e68592 [sys.catalog]: This master's current role is: LEADER
I20260929 22:52:47.616664 28636 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 22:52:47.617213 28636 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 22:52:47.618804 28636 catalog_manager.cc:1347] Generated new cluster ID: 28230963fa264659a5f158cbe3f10507
I20260929 22:52:47.618851 28636 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 29 22:52:47 dist-test-slave-98mk krb5kdc[28543](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790722367, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.147.254@KRBTEST.COM
I20260929 22:52:47.627938 28643 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:52:47.616544 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58928 (local address 127.14.147.254:44925)
0929 22:52:47.617056 (+ 512us) server_negotiation.cc:206] Beginning negotiation
0929 22:52:47.617064 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 22:52:47.617089 (+ 25us) server_negotiation.cc:407] Connection header received
0929 22:52:47.617191 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 22:52:47.617197 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 22:52:47.617242 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 22:52:47.617393 (+ 151us) server_negotiation.cc:226] Negotiated authn=SASL
0929 22:52:47.618067 (+ 674us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:52:47.619504 (+ 1437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:52:47.620580 (+ 1076us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:52:47.621278 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:52:47.621395 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 22:52:47.624228 (+ 2833us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 22:52:47.624234 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 22:52:47.624234 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 22:52:47.624248 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0929 22:52:47.626200 (+ 1952us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 22:52:47.626640 (+ 440us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 22:52:47.626643 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 22:52:47.626643 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 22:52:47.626743 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 22:52:47.626938 (+ 195us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 22:52:47.626939 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 22:52:47.626939 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 22:52:47.627243 (+ 304us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 22:52:47.627383 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0929 22:52:47.627539 (+ 156us) server_negotiation.cc:299] Negotiation successful
0929 22:52:47.627682 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":467,"thread_start_us":64,"threads_started":1}
I20260929 22:52:47.628911 14927 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 22:52:47.636202 28636 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 22:52:47.636726 28636 catalog_manager.cc:1504] Loading token signing keys...
I20260929 22:52:47.646631 28636 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0ed547e519a342a395a6ef2972e68592: Generated new TSK 0
I20260929 22:52:47.646819 28636 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 22:52:47.682937 14927 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhWOCLk/build/release/bin/kudu with pid 28556
2026-09-29T22:52:47Z chronyd exiting
[ OK ] SecurityITest.IPKICACertWebServerEndPoints (1264 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedIPKICert
Loading random data
Initializing database '/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790722213368738-14927-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 22:52:47 dist-test-slave-98mk krb5kdc[28651](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 22:52:47 dist-test-slave-98mk krb5kdc[28651](info): set up 2 sockets
Sep 29 22:52:47 dist-test-slave-98mk krb5kdc[28651](info): commencing operation
krb5kdc: starting...
W20260929 22:52:48.627959 14927 mini_kdc.cc:121] Time spent starting KDC: real 0.928s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790722213368738-14927-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790722213368738-14927-0/krb5kdc/test-user.keytab.
Sep 29 22:52:48 dist-test-slave-98mk krb5kdc[28651](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790722368, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T22:52:48Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T22:52:48Z Disabled control of system clock
WARNING: no policy specified for kudu/127.14.147.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.147.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.147.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790722213368738-14927-0/krb5kdc/kudu_127.14.147.254.keytab.
Entry for principal kudu/127.14.147.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790722213368738-14927-0/krb5kdc/kudu_127.14.147.254.keytab.
I20260929 22:52:48.738327 14927 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhWOCLk/build/release/bin/kudu
/tmp/dist-test-taskhWOCLk/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790722213368738-14927-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790722213368738-14927-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790722213368738-14927-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790722213368738-14927-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.147.254:35819
--webserver_interface=127.14.147.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790722213368738-14927-0/krb5kdc/kudu_127.14.147.254.keytab
--principal=kudu/127.14.147.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.147.212:43149
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.147.254:35819
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790722213368738-14927-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790722213368738-14927-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790722213368738-14927-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 22:52:48.825218 28664 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 22:52:48.825438 28664 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 22:52:48.825470 28664 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 22:52:48.826663 28664 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 22:52:48.826705 28664 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 22:52:48.826725 28664 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 22:52:48.826741 28664 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 22:52:48.826776 28664 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 22:52:48.828397 28664 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.147.212:43149
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790722213368738-14927-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790722213368738-14927-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.14.147.254:35819
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790722213368738-14927-0/krb5kdc/kudu_127.14.147.254.keytab
--principal=kudu/127.14.147.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.147.254:35819
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790722213368738-14927-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.14.147.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28664
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790722213368738-14927-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 21:58:54 UTC on ccycloud.quasar-xguoti.root.comops.site
build id 4681390
I20260929 22:52:48.828655 28664 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 22:52:48.828860 28664 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 22:52:48.834365 28669 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 22:52:48.834379 28670 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 22:52:48.834434 28664 server_base.cc:1034] running on GCE node
W20260929 22:52:48.834379 28672 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 22:52:48.834825 28664 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 22:52:48.835111 28664 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 22:52:48.836246 28664 hybrid_clock.cc:648] HybridClock initialized: now 1790722368836237 us; error 32 us; skew 500 ppm
Sep 29 22:52:48 dist-test-slave-98mk krb5kdc[28651](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790722368, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.147.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 22:52:48.838546 28664 init.cc:377] Logged in from keytab as kudu/127.14.147.254@KRBTEST.COM (short username kudu)
I20260929 22:52:48.838971 28664 webserver.cc:460] Webserver started at http://127.14.147.254:33785/ using document root <none> and password file <none>
I20260929 22:52:48.839190 28664 fs_manager.cc:359] Metadata directory not provided
I20260929 22:52:48.839237 28664 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 22:52:48.839371 28664 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 22:52:48.840444 28664 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790722213368738-14927-0/minicluster-data/master-0/data/instance:
uuid: "83dd29106e1946398652603810176095"
format_stamp: "Formatted at 2026-09-29 22:52:48 on dist-test-slave-98mk"
server_key: "dfa6a553431dbd9d21657950be5ff375"
server_key_iv: "dee84b8c1d392b6d5fef2d8ffe935993"
server_key_version: "encryptionkey@0"
I20260929 22:52:48.840772 28664 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790722213368738-14927-0/minicluster-data/master-0/wal/instance:
uuid: "83dd29106e1946398652603810176095"
format_stamp: "Formatted at 2026-09-29 22:52:48 on dist-test-slave-98mk"
server_key: "dfa6a553431dbd9d21657950be5ff375"
server_key_iv: "dee84b8c1d392b6d5fef2d8ffe935993"
server_key_version: "encryptionkey@0"
I20260929 22:52:48.842062 28664 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260929 22:52:48.842815 28679 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 22:52:48.842962 28664 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260929 22:52:48.843050 28664 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790722213368738-14927-0/minicluster-data/master-0/data,/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790722213368738-14927-0/minicluster-data/master-0/wal
uuid: "83dd29106e1946398652603810176095"
format_stamp: "Formatted at 2026-09-29 22:52:48 on dist-test-slave-98mk"
server_key: "dfa6a553431dbd9d21657950be5ff375"
server_key_iv: "dee84b8c1d392b6d5fef2d8ffe935993"
server_key_version: "encryptionkey@0"
I20260929 22:52:48.843117 28664 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790722213368738-14927-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790722213368738-14927-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790722213368738-14927-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 22:52:48.850595 28664 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 22:52:48.852573 28664 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 22:52:48.852708 28664 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 22:52:48.856459 28664 rpc_server.cc:307] RPC server started. Bound to: 127.14.147.254:35819
I20260929 22:52:48.856494 28731 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.147.254:35819 every 8 connection(s)
I20260929 22:52:48.856828 28664 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790722213368738-14927-0/minicluster-data/master-0/data/info.pb
I20260929 22:52:48.857751 28732 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 22:52:48.860133 28732 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 83dd29106e1946398652603810176095: Bootstrap starting.
I20260929 22:52:48.860705 28732 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 83dd29106e1946398652603810176095: Neither blocks nor log segments found. Creating new log.
I20260929 22:52:48.860960 28732 log.cc:824] T 00000000000000000000000000000000 P 83dd29106e1946398652603810176095: Log is configured to *not* fsync() on all Append() calls
I20260929 22:52:48.861673 28732 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 83dd29106e1946398652603810176095: No bootstrap required, opened a new log
I20260929 22:52:48.863214 14927 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhWOCLk/build/release/bin/kudu as pid 28664
I20260929 22:52:48.863322 14927 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790722213368738-14927-0/minicluster-data/master-0/wal/instance
I20260929 22:52:48.863524 14927 external_mini_cluster.cc:1428] Setting key f58c8f79693797b70b4f537a9475d95f
I20260929 22:52:48.863497 28732 raft_consensus.cc:348] T 00000000000000000000000000000000 P 83dd29106e1946398652603810176095 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "83dd29106e1946398652603810176095" member_type: VOTER last_known_addr { host: "127.14.147.254" port: 35819 } }
I20260929 22:52:48.863677 28732 raft_consensus.cc:374] T 00000000000000000000000000000000 P 83dd29106e1946398652603810176095 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 22:52:48.863718 28732 raft_consensus.cc:729] T 00000000000000000000000000000000 P 83dd29106e1946398652603810176095 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 83dd29106e1946398652603810176095, State: Initialized, Role: FOLLOWER
I20260929 22:52:48.863853 28732 consensus_queue.cc:260] T 00000000000000000000000000000000 P 83dd29106e1946398652603810176095 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "83dd29106e1946398652603810176095" member_type: VOTER last_known_addr { host: "127.14.147.254" port: 35819 } }
I20260929 22:52:48.863927 28732 raft_consensus.cc:388] T 00000000000000000000000000000000 P 83dd29106e1946398652603810176095 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 22:52:48.863956 28732 raft_consensus.cc:482] T 00000000000000000000000000000000 P 83dd29106e1946398652603810176095 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 22:52:48.863996 28732 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 83dd29106e1946398652603810176095 [term 0 FOLLOWER]: Advancing to term 1
I20260929 22:52:48.864624 28732 raft_consensus.cc:504] T 00000000000000000000000000000000 P 83dd29106e1946398652603810176095 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "83dd29106e1946398652603810176095" member_type: VOTER last_known_addr { host: "127.14.147.254" port: 35819 } }
I20260929 22:52:48.864720 28732 leader_election.cc:302] T 00000000000000000000000000000000 P 83dd29106e1946398652603810176095 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 83dd29106e1946398652603810176095; no voters:
I20260929 22:52:48.864871 28732 leader_election.cc:288] T 00000000000000000000000000000000 P 83dd29106e1946398652603810176095 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 22:52:48.864908 28737 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 83dd29106e1946398652603810176095 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 22:52:48.865006 28737 raft_consensus.cc:686] T 00000000000000000000000000000000 P 83dd29106e1946398652603810176095 [term 1 LEADER]: Becoming Leader. State: Replica: 83dd29106e1946398652603810176095, State: Running, Role: LEADER
I20260929 22:52:48.865085 28737 consensus_queue.cc:237] T 00000000000000000000000000000000 P 83dd29106e1946398652603810176095 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "83dd29106e1946398652603810176095" member_type: VOTER last_known_addr { host: "127.14.147.254" port: 35819 } }
I20260929 22:52:48.865123 28732 sys_catalog.cc:564] T 00000000000000000000000000000000 P 83dd29106e1946398652603810176095 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 22:52:48.865391 28737 sys_catalog.cc:455] T 00000000000000000000000000000000 P 83dd29106e1946398652603810176095 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 83dd29106e1946398652603810176095. Latest consensus state: current_term: 1 leader_uuid: "83dd29106e1946398652603810176095" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "83dd29106e1946398652603810176095" member_type: VOTER last_known_addr { host: "127.14.147.254" port: 35819 } } }
I20260929 22:52:48.865455 28737 sys_catalog.cc:458] T 00000000000000000000000000000000 P 83dd29106e1946398652603810176095 [sys.catalog]: This master's current role is: LEADER
I20260929 22:52:48.865446 28739 sys_catalog.cc:455] T 00000000000000000000000000000000 P 83dd29106e1946398652603810176095 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "83dd29106e1946398652603810176095" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "83dd29106e1946398652603810176095" member_type: VOTER last_known_addr { host: "127.14.147.254" port: 35819 } } }
I20260929 22:52:48.865680 28739 sys_catalog.cc:458] T 00000000000000000000000000000000 P 83dd29106e1946398652603810176095 [sys.catalog]: This master's current role is: LEADER
I20260929 22:52:48.866385 28744 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 22:52:48.866995 28744 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 22:52:48.868431 28744 catalog_manager.cc:1347] Generated new cluster ID: 56089a3d64ff483fa50c5dd79b352240
I20260929 22:52:48.868475 28744 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 29 22:52:48 dist-test-slave-98mk krb5kdc[28651](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790722368, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.147.254@KRBTEST.COM
I20260929 22:52:48.875790 28736 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:52:48.864001 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54096 (local address 127.14.147.254:35819)
0929 22:52:48.864174 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0929 22:52:48.864184 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 22:52:48.864218 (+ 34us) server_negotiation.cc:407] Connection header received
0929 22:52:48.864551 (+ 333us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 22:52:48.864557 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 22:52:48.864601 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 22:52:48.864721 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0929 22:52:48.865467 (+ 746us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:52:48.866849 (+ 1382us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:52:48.867938 (+ 1089us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:52:48.868518 (+ 580us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:52:48.868629 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 22:52:48.871574 (+ 2945us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 22:52:48.871578 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 22:52:48.871578 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 22:52:48.871591 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0929 22:52:48.873643 (+ 2052us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 22:52:48.874077 (+ 434us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 22:52:48.874081 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 22:52:48.874082 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 22:52:48.874228 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 22:52:48.874431 (+ 203us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 22:52:48.874432 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 22:52:48.874432 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 22:52:48.874776 (+ 344us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 22:52:48.874924 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0929 22:52:48.875379 (+ 455us) server_negotiation.cc:299] Negotiation successful
0929 22:52:48.875558 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":121,"thread_start_us":87,"threads_started":1}
I20260929 22:52:48.879740 28744 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 22:52:48.880430 28744 catalog_manager.cc:1504] Loading token signing keys...
I20260929 22:52:48.883939 28744 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 83dd29106e1946398652603810176095: Generated new TSK 0
I20260929 22:52:48.884127 28744 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.14.147.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.147.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.147.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790722213368738-14927-0/krb5kdc/kudu_127.14.147.193.keytab.
Entry for principal kudu/127.14.147.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790722213368738-14927-0/krb5kdc/kudu_127.14.147.193.keytab.
I20260929 22:52:48.907965 14927 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhWOCLk/build/release/bin/kudu
/tmp/dist-test-taskhWOCLk/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790722213368738-14927-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790722213368738-14927-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790722213368738-14927-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790722213368738-14927-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.147.193:0
--local_ip_for_outbound_sockets=127.14.147.193
--webserver_interface=127.14.147.193
--webserver_port=0
--tserver_master_addrs=127.14.147.254:35819
--keytab_file=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790722213368738-14927-0/krb5kdc/kudu_127.14.147.193.keytab
--principal=kudu/127.14.147.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.147.212:43149
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790722213368738-14927-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790722213368738-14927-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790722213368738-14927-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 22:52:48.989351 28758 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 22:52:48.989559 28758 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 22:52:48.989595 28758 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 22:52:48.990849 28758 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 22:52:48.990968 28758 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 22:52:48.991058 28758 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.147.193
I20260929 22:52:48.992851 28758 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.147.212:43149
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790722213368738-14927-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790722213368738-14927-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790722213368738-14927-0/krb5kdc/kudu_127.14.147.193.keytab
--principal=kudu/127.14.147.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.147.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790722213368738-14927-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.14.147.193
--webserver_port=0
--tserver_master_addrs=127.14.147.254:35819
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28758
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.147.193
--log_dir=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790722213368738-14927-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 21:58:54 UTC on ccycloud.quasar-xguoti.root.comops.site
build id 4681390
I20260929 22:52:48.993108 28758 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 22:52:48.993331 28758 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 22:52:48.998170 28763 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 22:52:48.998241 28758 server_base.cc:1034] running on GCE node
W20260929 22:52:48.998173 28764 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 22:52:48.998173 28766 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 22:52:48.998561 28758 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 22:52:48.998795 28758 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 22:52:48.999943 28758 hybrid_clock.cc:648] HybridClock initialized: now 1790722368999927 us; error 33 us; skew 500 ppm
Sep 29 22:52:48 dist-test-slave-98mk krb5kdc[28651](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790722368, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.147.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 22:52:49.002744 28758 init.cc:377] Logged in from keytab as kudu/127.14.147.193@KRBTEST.COM (short username kudu)
I20260929 22:52:49.003233 28758 webserver.cc:460] Webserver started at http://127.14.147.193:35563/ using document root <none> and password file <none>
I20260929 22:52:49.003513 28758 fs_manager.cc:359] Metadata directory not provided
I20260929 22:52:49.003562 28758 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 22:52:49.003692 28758 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 22:52:49.004789 28758 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790722213368738-14927-0/minicluster-data/ts-0/data/instance:
uuid: "43f3886bb1b44e4aabb3e8f71ff6f3b5"
format_stamp: "Formatted at 2026-09-29 22:52:49 on dist-test-slave-98mk"
server_key: "85d4f73702f5e7e081d2846fbdda5227"
server_key_iv: "933fc3cece9f429af97ebdb9a876d23e"
server_key_version: "encryptionkey@0"
I20260929 22:52:49.005113 28758 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790722213368738-14927-0/minicluster-data/ts-0/wal/instance:
uuid: "43f3886bb1b44e4aabb3e8f71ff6f3b5"
format_stamp: "Formatted at 2026-09-29 22:52:49 on dist-test-slave-98mk"
server_key: "85d4f73702f5e7e081d2846fbdda5227"
server_key_iv: "933fc3cece9f429af97ebdb9a876d23e"
server_key_version: "encryptionkey@0"
I20260929 22:52:49.006547 28758 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260929 22:52:49.007671 28773 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 22:52:49.007853 28758 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260929 22:52:49.007958 28758 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790722213368738-14927-0/minicluster-data/ts-0/data,/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790722213368738-14927-0/minicluster-data/ts-0/wal
uuid: "43f3886bb1b44e4aabb3e8f71ff6f3b5"
format_stamp: "Formatted at 2026-09-29 22:52:49 on dist-test-slave-98mk"
server_key: "85d4f73702f5e7e081d2846fbdda5227"
server_key_iv: "933fc3cece9f429af97ebdb9a876d23e"
server_key_version: "encryptionkey@0"
I20260929 22:52:49.008046 28758 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790722213368738-14927-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790722213368738-14927-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790722213368738-14927-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 22:52:49.020336 28758 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 22:52:49.022806 28758 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 22:52:49.023110 28758 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 22:52:49.023514 28758 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 22:52:49.023864 28758 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 22:52:49.023898 28758 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 22:52:49.023941 28758 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 22:52:49.023964 28758 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 22:52:49.029575 28758 rpc_server.cc:307] RPC server started. Bound to: 127.14.147.193:46859
I20260929 22:52:49.029633 28886 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.147.193:46859 every 8 connection(s)
I20260929 22:52:49.029942 28758 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790722213368738-14927-0/minicluster-data/ts-0/data/info.pb
I20260929 22:52:49.033141 14927 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhWOCLk/build/release/bin/kudu as pid 28758
I20260929 22:52:49.033249 14927 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790722213368738-14927-0/minicluster-data/ts-0/wal/instance
I20260929 22:52:49.033421 14927 external_mini_cluster.cc:1428] Setting key affedd1d28dfcdcaabf8ae4597f0780d
Sep 29 22:52:49 dist-test-slave-98mk krb5kdc[28651](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790722368, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.147.193@KRBTEST.COM for kudu/127.14.147.254@KRBTEST.COM
I20260929 22:52:49.039884 28736 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:52:49.030698 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.147.193:39645 (local address 127.14.147.254:35819)
0929 22:52:49.030781 (+ 83us) server_negotiation.cc:206] Beginning negotiation
0929 22:52:49.030784 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 22:52:49.030943 (+ 159us) server_negotiation.cc:407] Connection header received
0929 22:52:49.031435 (+ 492us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 22:52:49.031436 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 22:52:49.031451 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 22:52:49.031511 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0929 22:52:49.032407 (+ 896us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:52:49.032921 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:52:49.034124 (+ 1203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:52:49.034419 (+ 295us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:52:49.034487 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 22:52:49.036939 (+ 2452us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 22:52:49.036941 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 22:52:49.036942 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 22:52:49.036953 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0929 22:52:49.038332 (+ 1379us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 22:52:49.038819 (+ 487us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 22:52:49.038820 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 22:52:49.038820 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 22:52:49.038874 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 22:52:49.039144 (+ 270us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 22:52:49.039145 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 22:52:49.039145 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 22:52:49.039282 (+ 137us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 22:52:49.039334 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0929 22:52:49.039731 (+ 397us) server_negotiation.cc:299] Negotiation successful
0929 22:52:49.039812 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260929 22:52:49.040146 28889 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:52:49.030645 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.147.254:35819 (local address 127.14.147.193:39645)
0929 22:52:49.030823 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0929 22:52:49.030860 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0929 22:52:49.031342 (+ 482us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 22:52:49.031572 (+ 230us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 22:52:49.031576 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 22:52:49.032139 (+ 563us) client_negotiation.cc:189] Negotiated authn=SASL
0929 22:52:49.032334 (+ 195us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 22:52:49.032338 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 22:52:49.032987 (+ 649us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 22:52:49.032987 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 22:52:49.034033 (+ 1046us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 22:52:49.034036 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 22:52:49.034495 (+ 459us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 22:52:49.034496 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 22:52:49.034614 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 22:52:49.035137 (+ 523us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 22:52:49.035145 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0929 22:52:49.036805 (+ 1660us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 22:52:49.038419 (+ 1614us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 22:52:49.038436 (+ 17us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 22:52:49.038439 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 22:52:49.038744 (+ 305us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 22:52:49.038938 (+ 194us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 22:52:49.038939 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 22:52:49.038939 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 22:52:49.039083 (+ 144us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 22:52:49.039352 (+ 269us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 22:52:49.039359 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 22:52:49.039585 (+ 226us) client_negotiation.cc:769] Sending connection context
0929 22:52:49.039712 (+ 127us) client_negotiation.cc:240] Negotiation successful
0929 22:52:49.039900 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":139,"thread_start_us":91,"threads_started":1}
I20260929 22:52:49.040725 28887 heartbeater.cc:347] Connected to a master server at 127.14.147.254:35819
I20260929 22:52:49.040835 28887 heartbeater.cc:464] Registering TS with master...
I20260929 22:52:49.041049 28887 heartbeater.cc:511] Master 127.14.147.254:35819 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.14.147.194@KRBTEST.COM; defaulting to no policy
I20260929 22:52:49.041852 28696 ts_manager.cc:194] Registered new tserver with Master: 43f3886bb1b44e4aabb3e8f71ff6f3b5 (127.14.147.193:46859)
I20260929 22:52:49.042824 28696 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.147.193@KRBTEST.COM'} at 127.14.147.193:39645
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.147.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.147.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790722213368738-14927-0/krb5kdc/kudu_127.14.147.194.keytab.
Entry for principal kudu/127.14.147.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790722213368738-14927-0/krb5kdc/kudu_127.14.147.194.keytab.
I20260929 22:52:49.057602 14927 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhWOCLk/build/release/bin/kudu
/tmp/dist-test-taskhWOCLk/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790722213368738-14927-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790722213368738-14927-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790722213368738-14927-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790722213368738-14927-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.147.194:0
--local_ip_for_outbound_sockets=127.14.147.194
--webserver_interface=127.14.147.194
--webserver_port=0
--tserver_master_addrs=127.14.147.254:35819
--keytab_file=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790722213368738-14927-0/krb5kdc/kudu_127.14.147.194.keytab
--principal=kudu/127.14.147.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.147.212:43149
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790722213368738-14927-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790722213368738-14927-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790722213368738-14927-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 22:52:49.138185 28892 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 22:52:49.138403 28892 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 22:52:49.138427 28892 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 22:52:49.139631 28892 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 22:52:49.139671 28892 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 22:52:49.139698 28892 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.147.194
I20260929 22:52:49.141222 28892 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.147.212:43149
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790722213368738-14927-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790722213368738-14927-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790722213368738-14927-0/krb5kdc/kudu_127.14.147.194.keytab
--principal=kudu/127.14.147.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.147.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790722213368738-14927-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.14.147.194
--webserver_port=0
--tserver_master_addrs=127.14.147.254:35819
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28892
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.147.194
--log_dir=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790722213368738-14927-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 21:58:54 UTC on ccycloud.quasar-xguoti.root.comops.site
build id 4681390
I20260929 22:52:49.141436 28892 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 22:52:49.141636 28892 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 22:52:49.146530 28897 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 22:52:49.146541 28900 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 22:52:49.146554 28898 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 22:52:49.146782 28892 server_base.cc:1034] running on GCE node
I20260929 22:52:49.146958 28892 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 22:52:49.147178 28892 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 22:52:49.148360 28892 hybrid_clock.cc:648] HybridClock initialized: now 1790722369148344 us; error 26 us; skew 500 ppm
Sep 29 22:52:49 dist-test-slave-98mk krb5kdc[28651](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790722369, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.147.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 22:52:49.150807 28892 init.cc:377] Logged in from keytab as kudu/127.14.147.194@KRBTEST.COM (short username kudu)
I20260929 22:52:49.151274 28892 webserver.cc:460] Webserver started at http://127.14.147.194:41865/ using document root <none> and password file <none>
I20260929 22:52:49.151527 28892 fs_manager.cc:359] Metadata directory not provided
I20260929 22:52:49.151575 28892 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 22:52:49.151700 28892 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 22:52:49.152776 28892 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790722213368738-14927-0/minicluster-data/ts-1/data/instance:
uuid: "ae4204ae8cb54f4f8f9a085ca5976542"
format_stamp: "Formatted at 2026-09-29 22:52:49 on dist-test-slave-98mk"
server_key: "2e9a6145dda39dc3abf6c3e9ece36713"
server_key_iv: "e70bb44c2b3731c6d82df39796783df4"
server_key_version: "encryptionkey@0"
I20260929 22:52:49.153098 28892 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790722213368738-14927-0/minicluster-data/ts-1/wal/instance:
uuid: "ae4204ae8cb54f4f8f9a085ca5976542"
format_stamp: "Formatted at 2026-09-29 22:52:49 on dist-test-slave-98mk"
server_key: "2e9a6145dda39dc3abf6c3e9ece36713"
server_key_iv: "e70bb44c2b3731c6d82df39796783df4"
server_key_version: "encryptionkey@0"
I20260929 22:52:49.154441 28892 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260929 22:52:49.155202 28907 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 22:52:49.155530 28892 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260929 22:52:49.155732 28892 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790722213368738-14927-0/minicluster-data/ts-1/data,/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790722213368738-14927-0/minicluster-data/ts-1/wal
uuid: "ae4204ae8cb54f4f8f9a085ca5976542"
format_stamp: "Formatted at 2026-09-29 22:52:49 on dist-test-slave-98mk"
server_key: "2e9a6145dda39dc3abf6c3e9ece36713"
server_key_iv: "e70bb44c2b3731c6d82df39796783df4"
server_key_version: "encryptionkey@0"
I20260929 22:52:49.155825 28892 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790722213368738-14927-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790722213368738-14927-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790722213368738-14927-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 22:52:49.169299 28892 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 22:52:49.171401 28892 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 22:52:49.171576 28892 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 22:52:49.171808 28892 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 22:52:49.172127 28892 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 22:52:49.172163 28892 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 22:52:49.172191 28892 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 22:52:49.172209 28892 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 22:52:49.177851 28892 rpc_server.cc:307] RPC server started. Bound to: 127.14.147.194:46603
I20260929 22:52:49.177928 29020 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.147.194:46603 every 8 connection(s)
I20260929 22:52:49.178229 28892 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790722213368738-14927-0/minicluster-data/ts-1/data/info.pb
I20260929 22:52:49.182642 14927 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhWOCLk/build/release/bin/kudu as pid 28892
I20260929 22:52:49.182745 14927 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790722213368738-14927-0/minicluster-data/ts-1/wal/instance
I20260929 22:52:49.182910 14927 external_mini_cluster.cc:1428] Setting key 04b04b6ff789b7e981dce9c3c6c94d39
Sep 29 22:52:49 dist-test-slave-98mk krb5kdc[28651](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790722369, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.147.194@KRBTEST.COM for kudu/127.14.147.254@KRBTEST.COM
I20260929 22:52:49.188261 28736 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:52:49.178927 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.147.194:52865 (local address 127.14.147.254:35819)
0929 22:52:49.178978 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0929 22:52:49.178980 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 22:52:49.179177 (+ 197us) server_negotiation.cc:407] Connection header received
0929 22:52:49.179714 (+ 537us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 22:52:49.179715 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 22:52:49.179724 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 22:52:49.179783 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0929 22:52:49.180712 (+ 929us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:52:49.181249 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:52:49.182265 (+ 1016us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:52:49.182507 (+ 242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:52:49.182587 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 22:52:49.185299 (+ 2712us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 22:52:49.185301 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 22:52:49.185301 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 22:52:49.185311 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0929 22:52:49.186661 (+ 1350us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 22:52:49.187164 (+ 503us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 22:52:49.187165 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 22:52:49.187166 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 22:52:49.187239 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 22:52:49.187546 (+ 307us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 22:52:49.187546 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 22:52:49.187546 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 22:52:49.187657 (+ 111us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 22:52:49.187705 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0929 22:52:49.188095 (+ 390us) server_negotiation.cc:299] Negotiation successful
0929 22:52:49.188183 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260929 22:52:49.188472 29023 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:52:49.178891 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.147.254:35819 (local address 127.14.147.194:52865)
0929 22:52:49.179054 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0929 22:52:49.179099 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0929 22:52:49.179628 (+ 529us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 22:52:49.179842 (+ 214us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 22:52:49.179845 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 22:52:49.180405 (+ 560us) client_negotiation.cc:189] Negotiated authn=SASL
0929 22:52:49.180605 (+ 200us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 22:52:49.180610 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 22:52:49.181355 (+ 745us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 22:52:49.181356 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 22:52:49.182173 (+ 817us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 22:52:49.182175 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 22:52:49.182701 (+ 526us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 22:52:49.182703 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 22:52:49.182960 (+ 257us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 22:52:49.183719 (+ 759us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 22:52:49.183732 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0929 22:52:49.185190 (+ 1458us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 22:52:49.186750 (+ 1560us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 22:52:49.186758 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 22:52:49.186761 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 22:52:49.187061 (+ 300us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 22:52:49.187307 (+ 246us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 22:52:49.187307 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 22:52:49.187308 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 22:52:49.187482 (+ 174us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 22:52:49.187712 (+ 230us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 22:52:49.187719 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 22:52:49.187905 (+ 186us) client_negotiation.cc:769] Sending connection context
0929 22:52:49.188024 (+ 119us) client_negotiation.cc:240] Negotiation successful
0929 22:52:49.188213 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":70,"threads_started":1}
I20260929 22:52:49.189116 29021 heartbeater.cc:347] Connected to a master server at 127.14.147.254:35819
I20260929 22:52:49.189213 29021 heartbeater.cc:464] Registering TS with master...
I20260929 22:52:49.189393 29021 heartbeater.cc:511] Master 127.14.147.254:35819 requested a full tablet report, sending...
I20260929 22:52:49.189769 28696 ts_manager.cc:194] Registered new tserver with Master: ae4204ae8cb54f4f8f9a085ca5976542 (127.14.147.194:46603)
I20260929 22:52:49.190148 28696 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.147.194@KRBTEST.COM'} at 127.14.147.194:52865
WARNING: no policy specified for kudu/127.14.147.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.147.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.147.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790722213368738-14927-0/krb5kdc/kudu_127.14.147.195.keytab.
Entry for principal kudu/127.14.147.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790722213368738-14927-0/krb5kdc/kudu_127.14.147.195.keytab.
I20260929 22:52:49.206142 14927 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhWOCLk/build/release/bin/kudu
/tmp/dist-test-taskhWOCLk/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790722213368738-14927-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790722213368738-14927-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790722213368738-14927-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790722213368738-14927-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.147.195:0
--local_ip_for_outbound_sockets=127.14.147.195
--webserver_interface=127.14.147.195
--webserver_port=0
--tserver_master_addrs=127.14.147.254:35819
--keytab_file=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790722213368738-14927-0/krb5kdc/kudu_127.14.147.195.keytab
--principal=kudu/127.14.147.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.147.212:43149
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790722213368738-14927-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790722213368738-14927-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790722213368738-14927-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 22:52:49.287557 29026 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 22:52:49.287793 29026 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 22:52:49.287828 29026 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 22:52:49.289070 29026 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 22:52:49.289119 29026 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 22:52:49.289157 29026 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.147.195
I20260929 22:52:49.290706 29026 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.147.212:43149
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790722213368738-14927-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790722213368738-14927-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790722213368738-14927-0/krb5kdc/kudu_127.14.147.195.keytab
--principal=kudu/127.14.147.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.147.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790722213368738-14927-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.14.147.195
--webserver_port=0
--tserver_master_addrs=127.14.147.254:35819
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29026
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.147.195
--log_dir=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790722213368738-14927-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 21:58:54 UTC on ccycloud.quasar-xguoti.root.comops.site
build id 4681390
I20260929 22:52:49.290958 29026 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 22:52:49.291180 29026 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 22:52:49.296010 29031 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 22:52:49.296103 29026 server_base.cc:1034] running on GCE node
W20260929 22:52:49.296005 29034 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 22:52:49.296186 29032 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 22:52:49.296377 29026 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 22:52:49.296607 29026 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 22:52:49.297761 29026 hybrid_clock.cc:648] HybridClock initialized: now 1790722369297747 us; error 30 us; skew 500 ppm
Sep 29 22:52:49 dist-test-slave-98mk krb5kdc[28651](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790722369, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.147.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 22:52:49.300320 29026 init.cc:377] Logged in from keytab as kudu/127.14.147.195@KRBTEST.COM (short username kudu)
I20260929 22:52:49.300777 29026 webserver.cc:460] Webserver started at http://127.14.147.195:32777/ using document root <none> and password file <none>
I20260929 22:52:49.301002 29026 fs_manager.cc:359] Metadata directory not provided
I20260929 22:52:49.301050 29026 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 22:52:49.301170 29026 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 22:52:49.302202 29026 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790722213368738-14927-0/minicluster-data/ts-2/data/instance:
uuid: "80e8234059154efbbc9417e8b4a71810"
format_stamp: "Formatted at 2026-09-29 22:52:49 on dist-test-slave-98mk"
server_key: "1b6941b1003aef74534e2284c46deca2"
server_key_iv: "4e9b88308cdcc5d800a9607e592d76ae"
server_key_version: "encryptionkey@0"
I20260929 22:52:49.302516 29026 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790722213368738-14927-0/minicluster-data/ts-2/wal/instance:
uuid: "80e8234059154efbbc9417e8b4a71810"
format_stamp: "Formatted at 2026-09-29 22:52:49 on dist-test-slave-98mk"
server_key: "1b6941b1003aef74534e2284c46deca2"
server_key_iv: "4e9b88308cdcc5d800a9607e592d76ae"
server_key_version: "encryptionkey@0"
I20260929 22:52:49.303782 29026 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260929 22:52:49.304524 29041 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 22:52:49.304669 29026 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260929 22:52:49.304749 29026 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790722213368738-14927-0/minicluster-data/ts-2/data,/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790722213368738-14927-0/minicluster-data/ts-2/wal
uuid: "80e8234059154efbbc9417e8b4a71810"
format_stamp: "Formatted at 2026-09-29 22:52:49 on dist-test-slave-98mk"
server_key: "1b6941b1003aef74534e2284c46deca2"
server_key_iv: "4e9b88308cdcc5d800a9607e592d76ae"
server_key_version: "encryptionkey@0"
I20260929 22:52:49.304809 29026 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790722213368738-14927-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790722213368738-14927-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790722213368738-14927-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 22:52:49.313326 29026 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 22:52:49.315032 29026 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 22:52:49.315157 29026 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 22:52:49.315347 29026 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 22:52:49.315650 29026 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 22:52:49.315682 29026 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 22:52:49.315711 29026 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 22:52:49.315730 29026 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 22:52:49.321115 29026 rpc_server.cc:307] RPC server started. Bound to: 127.14.147.195:41003
I20260929 22:52:49.321173 29154 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.147.195:41003 every 8 connection(s)
I20260929 22:52:49.321458 29026 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790722213368738-14927-0/minicluster-data/ts-2/data/info.pb
Sep 29 22:52:49 dist-test-slave-98mk krb5kdc[28651](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790722369, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.147.195@KRBTEST.COM for kudu/127.14.147.254@KRBTEST.COM
I20260929 22:52:49.329649 28736 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:52:49.322124 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.147.195:38265 (local address 127.14.147.254:35819)
0929 22:52:49.322189 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0929 22:52:49.322192 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 22:52:49.322363 (+ 171us) server_negotiation.cc:407] Connection header received
0929 22:52:49.322860 (+ 497us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 22:52:49.322861 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 22:52:49.322872 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 22:52:49.322919 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0929 22:52:49.323879 (+ 960us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:52:49.324376 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:52:49.325318 (+ 942us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:52:49.325510 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:52:49.325563 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 22:52:49.327549 (+ 1986us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 22:52:49.327551 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 22:52:49.327551 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 22:52:49.327558 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0929 22:52:49.328576 (+ 1018us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 22:52:49.328906 (+ 330us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 22:52:49.328907 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 22:52:49.328907 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 22:52:49.328947 (+ 40us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 22:52:49.329111 (+ 164us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 22:52:49.329112 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 22:52:49.329112 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 22:52:49.329204 (+ 92us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 22:52:49.329249 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0929 22:52:49.329528 (+ 279us) server_negotiation.cc:299] Negotiation successful
0929 22:52:49.329583 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260929 22:52:49.329861 29157 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:52:49.322081 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.147.254:35819 (local address 127.14.147.195:38265)
0929 22:52:49.322250 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0929 22:52:49.322284 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0929 22:52:49.322765 (+ 481us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 22:52:49.322997 (+ 232us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 22:52:49.323000 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 22:52:49.323600 (+ 600us) client_negotiation.cc:189] Negotiated authn=SASL
0929 22:52:49.323793 (+ 193us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 22:52:49.323797 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 22:52:49.324458 (+ 661us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 22:52:49.324459 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 22:52:49.325236 (+ 777us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 22:52:49.325238 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 22:52:49.325576 (+ 338us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 22:52:49.325577 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 22:52:49.325695 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 22:52:49.326150 (+ 455us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 22:52:49.326158 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0929 22:52:49.327454 (+ 1296us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 22:52:49.328644 (+ 1190us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 22:52:49.328650 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 22:52:49.328652 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 22:52:49.328857 (+ 205us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 22:52:49.328987 (+ 130us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 22:52:49.328988 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 22:52:49.328988 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 22:52:49.329072 (+ 84us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 22:52:49.329281 (+ 209us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 22:52:49.329288 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 22:52:49.329413 (+ 125us) client_negotiation.cc:769] Sending connection context
0929 22:52:49.329510 (+ 97us) client_negotiation.cc:240] Negotiation successful
0929 22:52:49.329635 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":131,"thread_start_us":77,"threads_started":1}
I20260929 22:52:49.330327 29155 heartbeater.cc:347] Connected to a master server at 127.14.147.254:35819
I20260929 22:52:49.330418 29155 heartbeater.cc:464] Registering TS with master...
I20260929 22:52:49.330587 29155 heartbeater.cc:511] Master 127.14.147.254:35819 requested a full tablet report, sending...
I20260929 22:52:49.330953 28696 ts_manager.cc:194] Registered new tserver with Master: 80e8234059154efbbc9417e8b4a71810 (127.14.147.195:41003)
I20260929 22:52:49.331239 14927 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhWOCLk/build/release/bin/kudu as pid 29026
I20260929 22:52:49.331329 14927 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790722213368738-14927-0/minicluster-data/ts-2/wal/instance
I20260929 22:52:49.331404 28696 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.147.195@KRBTEST.COM'} at 127.14.147.195:38265
I20260929 22:52:49.331534 14927 external_mini_cluster.cc:1428] Setting key 31436b9b2a10c55e796408aeee47c688
I20260929 22:52:49.332340 14927 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 22:52:49.338696 28736 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:52:49.333558 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54110 (local address 127.14.147.254:35819)
0929 22:52:49.333601 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0929 22:52:49.333603 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 22:52:49.333709 (+ 106us) server_negotiation.cc:407] Connection header received
0929 22:52:49.333766 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 22:52:49.333766 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 22:52:49.333774 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 22:52:49.333809 (+ 35us) server_negotiation.cc:226] Negotiated authn=SASL
0929 22:52:49.334280 (+ 471us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:52:49.334742 (+ 462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:52:49.335391 (+ 649us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:52:49.335626 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:52:49.335674 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 22:52:49.336690 (+ 1016us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 22:52:49.336691 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 22:52:49.336691 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 22:52:49.336698 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0929 22:52:49.337912 (+ 1214us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 22:52:49.338149 (+ 237us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 22:52:49.338150 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 22:52:49.338150 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 22:52:49.338193 (+ 43us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 22:52:49.338307 (+ 114us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 22:52:49.338307 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 22:52:49.338307 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 22:52:49.338438 (+ 131us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 22:52:49.338511 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0929 22:52:49.338574 (+ 63us) server_negotiation.cc:299] Negotiation successful
0929 22:52:49.338630 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260929 22:52:49.340075 28696 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54110:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 22:52:49.341593 28696 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 22:52:49.347743 29166 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:52:49.343789 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53942 (local address 127.14.147.194:46603)
0929 22:52:49.343952 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0929 22:52:49.343956 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 22:52:49.343968 (+ 12us) server_negotiation.cc:407] Connection header received
0929 22:52:49.344273 (+ 305us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 22:52:49.344276 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 22:52:49.344290 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 22:52:49.344357 (+ 67us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 22:52:49.344559 (+ 202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:52:49.345254 (+ 695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:52:49.346722 (+ 1468us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:52:49.347306 (+ 584us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:52:49.347373 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 22:52:49.347459 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0929 22:52:49.347605 (+ 146us) server_negotiation.cc:299] Negotiation successful
0929 22:52:49.347652 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":132,"thread_start_us":67,"threads_started":1}
I20260929 22:52:49.347774 29164 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:52:49.343634 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.147.194:46603 (local address 127.0.0.1:53942)
0929 22:52:49.343918 (+ 284us) negotiation.cc:107] Waiting for socket to connect
0929 22:52:49.343928 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0929 22:52:49.344173 (+ 245us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 22:52:49.344428 (+ 255us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 22:52:49.344429 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 22:52:49.344431 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 22:52:49.344485 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 22:52:49.344487 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 22:52:49.345343 (+ 856us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 22:52:49.345344 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 22:52:49.346641 (+ 1297us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 22:52:49.346644 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 22:52:49.347383 (+ 739us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 22:52:49.347384 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 22:52:49.347504 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 22:52:49.347520 (+ 16us) client_negotiation.cc:769] Sending connection context
0929 22:52:49.347584 (+ 64us) client_negotiation.cc:240] Negotiation successful
0929 22:52:49.347670 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":257,"mutex_wait_us":2,"thread_start_us":74,"threads_started":1}
I20260929 22:52:49.348059 29167 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:52:49.343820 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.147.195:41003 (local address 127.0.0.1:34162)
0929 22:52:49.344294 (+ 474us) negotiation.cc:107] Waiting for socket to connect
0929 22:52:49.344303 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0929 22:52:49.344353 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 22:52:49.344600 (+ 247us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 22:52:49.344601 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 22:52:49.344603 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 22:52:49.344641 (+ 38us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 22:52:49.344642 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 22:52:49.345870 (+ 1228us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 22:52:49.345871 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 22:52:49.346793 (+ 922us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 22:52:49.346795 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 22:52:49.347779 (+ 984us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 22:52:49.347780 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 22:52:49.347896 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 22:52:49.347901 (+ 5us) client_negotiation.cc:769] Sending connection context
0929 22:52:49.347944 (+ 43us) client_negotiation.cc:240] Negotiation successful
0929 22:52:49.347986 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":456,"thread_start_us":67,"threads_started":1}
I20260929 22:52:49.348295 29168 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:52:49.344019 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34162 (local address 127.14.147.195:41003)
0929 22:52:49.344219 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0929 22:52:49.344224 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 22:52:49.344349 (+ 125us) server_negotiation.cc:407] Connection header received
0929 22:52:49.344439 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 22:52:49.344441 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 22:52:49.344457 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 22:52:49.344522 (+ 65us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 22:52:49.344710 (+ 188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:52:49.345767 (+ 1057us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:52:49.346871 (+ 1104us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:52:49.347621 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:52:49.347687 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 22:52:49.347736 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0929 22:52:49.348167 (+ 431us) server_negotiation.cc:299] Negotiation successful
0929 22:52:49.348213 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":148,"thread_start_us":86,"threads_started":1}
I20260929 22:52:49.348589 28955 tablet_service.cc:1467] Processing CreateTablet for tablet e4c4c66b006c4d45a1561ab37d0e3cb6 (DEFAULT_TABLE table=test-table [id=58881d3f259544f09bb6295e9259719f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 22:52:49.348942 28955 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e4c4c66b006c4d45a1561ab37d0e3cb6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 22:52:49.348958 29089 tablet_service.cc:1467] Processing CreateTablet for tablet e4c4c66b006c4d45a1561ab37d0e3cb6 (DEFAULT_TABLE table=test-table [id=58881d3f259544f09bb6295e9259719f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 22:52:49.349166 29163 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:52:49.343633 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.147.193:46859 (local address 127.0.0.1:58728)
0929 22:52:49.343803 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0929 22:52:49.343843 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0929 22:52:49.343907 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 22:52:49.344226 (+ 319us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 22:52:49.344229 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 22:52:49.344232 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 22:52:49.344306 (+ 74us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 22:52:49.344309 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 22:52:49.345760 (+ 1451us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 22:52:49.345761 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 22:52:49.346815 (+ 1054us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 22:52:49.346817 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 22:52:49.348945 (+ 2128us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 22:52:49.348946 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 22:52:49.349012 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 22:52:49.349016 (+ 4us) client_negotiation.cc:769] Sending connection context
0929 22:52:49.349063 (+ 47us) client_negotiation.cc:240] Negotiation successful
0929 22:52:49.349105 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":147,"spinlock_wait_cycles":896,"thread_start_us":59,"threads_started":1}
I20260929 22:52:49.349287 29089 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e4c4c66b006c4d45a1561ab37d0e3cb6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 22:52:49.349457 29165 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:52:49.343726 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58728 (local address 127.14.147.193:46859)
0929 22:52:49.344039 (+ 313us) server_negotiation.cc:206] Beginning negotiation
0929 22:52:49.344043 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 22:52:49.344052 (+ 9us) server_negotiation.cc:407] Connection header received
0929 22:52:49.344090 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 22:52:49.344092 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 22:52:49.344109 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 22:52:49.344192 (+ 83us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 22:52:49.344517 (+ 325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:52:49.345651 (+ 1134us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:52:49.346892 (+ 1241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:52:49.348860 (+ 1968us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:52:49.348945 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 22:52:49.349094 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0929 22:52:49.349188 (+ 94us) server_negotiation.cc:299] Negotiation successful
0929 22:52:49.349384 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":284,"thread_start_us":96,"threads_started":1}
I20260929 22:52:49.350054 28821 tablet_service.cc:1467] Processing CreateTablet for tablet e4c4c66b006c4d45a1561ab37d0e3cb6 (DEFAULT_TABLE table=test-table [id=58881d3f259544f09bb6295e9259719f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 22:52:49.350354 28821 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e4c4c66b006c4d45a1561ab37d0e3cb6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 22:52:49.351949 29169 tablet_bootstrap.cc:492] T e4c4c66b006c4d45a1561ab37d0e3cb6 P 80e8234059154efbbc9417e8b4a71810: Bootstrap starting.
I20260929 22:52:49.352234 29170 tablet_bootstrap.cc:492] T e4c4c66b006c4d45a1561ab37d0e3cb6 P ae4204ae8cb54f4f8f9a085ca5976542: Bootstrap starting.
I20260929 22:52:49.352576 29169 tablet_bootstrap.cc:654] T e4c4c66b006c4d45a1561ab37d0e3cb6 P 80e8234059154efbbc9417e8b4a71810: Neither blocks nor log segments found. Creating new log.
I20260929 22:52:49.352831 29170 tablet_bootstrap.cc:654] T e4c4c66b006c4d45a1561ab37d0e3cb6 P ae4204ae8cb54f4f8f9a085ca5976542: Neither blocks nor log segments found. Creating new log.
I20260929 22:52:49.353106 29169 log.cc:824] T e4c4c66b006c4d45a1561ab37d0e3cb6 P 80e8234059154efbbc9417e8b4a71810: Log is configured to *not* fsync() on all Append() calls
I20260929 22:52:49.353235 29171 tablet_bootstrap.cc:492] T e4c4c66b006c4d45a1561ab37d0e3cb6 P 43f3886bb1b44e4aabb3e8f71ff6f3b5: Bootstrap starting.
I20260929 22:52:49.353317 29170 log.cc:824] T e4c4c66b006c4d45a1561ab37d0e3cb6 P ae4204ae8cb54f4f8f9a085ca5976542: Log is configured to *not* fsync() on all Append() calls
I20260929 22:52:49.353803 29169 tablet_bootstrap.cc:492] T e4c4c66b006c4d45a1561ab37d0e3cb6 P 80e8234059154efbbc9417e8b4a71810: No bootstrap required, opened a new log
I20260929 22:52:49.353847 29171 tablet_bootstrap.cc:654] T e4c4c66b006c4d45a1561ab37d0e3cb6 P 43f3886bb1b44e4aabb3e8f71ff6f3b5: Neither blocks nor log segments found. Creating new log.
I20260929 22:52:49.353864 29169 ts_tablet_manager.cc:1397] T e4c4c66b006c4d45a1561ab37d0e3cb6 P 80e8234059154efbbc9417e8b4a71810: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260929 22:52:49.354007 29170 tablet_bootstrap.cc:492] T e4c4c66b006c4d45a1561ab37d0e3cb6 P ae4204ae8cb54f4f8f9a085ca5976542: No bootstrap required, opened a new log
I20260929 22:52:49.354079 29170 ts_tablet_manager.cc:1397] T e4c4c66b006c4d45a1561ab37d0e3cb6 P ae4204ae8cb54f4f8f9a085ca5976542: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260929 22:52:49.354422 29171 log.cc:824] T e4c4c66b006c4d45a1561ab37d0e3cb6 P 43f3886bb1b44e4aabb3e8f71ff6f3b5: Log is configured to *not* fsync() on all Append() calls
I20260929 22:52:49.355140 29171 tablet_bootstrap.cc:492] T e4c4c66b006c4d45a1561ab37d0e3cb6 P 43f3886bb1b44e4aabb3e8f71ff6f3b5: No bootstrap required, opened a new log
I20260929 22:52:49.355218 29171 ts_tablet_manager.cc:1397] T e4c4c66b006c4d45a1561ab37d0e3cb6 P 43f3886bb1b44e4aabb3e8f71ff6f3b5: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260929 22:52:49.355607 29169 raft_consensus.cc:348] T e4c4c66b006c4d45a1561ab37d0e3cb6 P 80e8234059154efbbc9417e8b4a71810 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae4204ae8cb54f4f8f9a085ca5976542" member_type: VOTER last_known_addr { host: "127.14.147.194" port: 46603 } } peers { permanent_uuid: "43f3886bb1b44e4aabb3e8f71ff6f3b5" member_type: VOTER last_known_addr { host: "127.14.147.193" port: 46859 } } peers { permanent_uuid: "80e8234059154efbbc9417e8b4a71810" member_type: VOTER last_known_addr { host: "127.14.147.195" port: 41003 } }
I20260929 22:52:49.355741 29169 raft_consensus.cc:374] T e4c4c66b006c4d45a1561ab37d0e3cb6 P 80e8234059154efbbc9417e8b4a71810 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 22:52:49.355772 29169 raft_consensus.cc:729] T e4c4c66b006c4d45a1561ab37d0e3cb6 P 80e8234059154efbbc9417e8b4a71810 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 80e8234059154efbbc9417e8b4a71810, State: Initialized, Role: FOLLOWER
I20260929 22:52:49.355881 29169 consensus_queue.cc:260] T e4c4c66b006c4d45a1561ab37d0e3cb6 P 80e8234059154efbbc9417e8b4a71810 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae4204ae8cb54f4f8f9a085ca5976542" member_type: VOTER last_known_addr { host: "127.14.147.194" port: 46603 } } peers { permanent_uuid: "43f3886bb1b44e4aabb3e8f71ff6f3b5" member_type: VOTER last_known_addr { host: "127.14.147.193" port: 46859 } } peers { permanent_uuid: "80e8234059154efbbc9417e8b4a71810" member_type: VOTER last_known_addr { host: "127.14.147.195" port: 41003 } }
I20260929 22:52:49.355968 29170 raft_consensus.cc:348] T e4c4c66b006c4d45a1561ab37d0e3cb6 P ae4204ae8cb54f4f8f9a085ca5976542 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae4204ae8cb54f4f8f9a085ca5976542" member_type: VOTER last_known_addr { host: "127.14.147.194" port: 46603 } } peers { permanent_uuid: "43f3886bb1b44e4aabb3e8f71ff6f3b5" member_type: VOTER last_known_addr { host: "127.14.147.193" port: 46859 } } peers { permanent_uuid: "80e8234059154efbbc9417e8b4a71810" member_type: VOTER last_known_addr { host: "127.14.147.195" port: 41003 } }
I20260929 22:52:49.356127 29170 raft_consensus.cc:374] T e4c4c66b006c4d45a1561ab37d0e3cb6 P ae4204ae8cb54f4f8f9a085ca5976542 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 22:52:49.356156 29169 ts_tablet_manager.cc:1428] T e4c4c66b006c4d45a1561ab37d0e3cb6 P 80e8234059154efbbc9417e8b4a71810: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260929 22:52:49.356168 29170 raft_consensus.cc:729] T e4c4c66b006c4d45a1561ab37d0e3cb6 P ae4204ae8cb54f4f8f9a085ca5976542 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ae4204ae8cb54f4f8f9a085ca5976542, State: Initialized, Role: FOLLOWER
I20260929 22:52:49.356272 29155 heartbeater.cc:503] Master 127.14.147.254:35819 was elected leader, sending a full tablet report...
I20260929 22:52:49.356267 29170 consensus_queue.cc:260] T e4c4c66b006c4d45a1561ab37d0e3cb6 P ae4204ae8cb54f4f8f9a085ca5976542 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae4204ae8cb54f4f8f9a085ca5976542" member_type: VOTER last_known_addr { host: "127.14.147.194" port: 46603 } } peers { permanent_uuid: "43f3886bb1b44e4aabb3e8f71ff6f3b5" member_type: VOTER last_known_addr { host: "127.14.147.193" port: 46859 } } peers { permanent_uuid: "80e8234059154efbbc9417e8b4a71810" member_type: VOTER last_known_addr { host: "127.14.147.195" port: 41003 } }
I20260929 22:52:49.356508 29170 ts_tablet_manager.cc:1428] T e4c4c66b006c4d45a1561ab37d0e3cb6 P ae4204ae8cb54f4f8f9a085ca5976542: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260929 22:52:49.356622 29021 heartbeater.cc:503] Master 127.14.147.254:35819 was elected leader, sending a full tablet report...
I20260929 22:52:49.357292 29171 raft_consensus.cc:348] T e4c4c66b006c4d45a1561ab37d0e3cb6 P 43f3886bb1b44e4aabb3e8f71ff6f3b5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae4204ae8cb54f4f8f9a085ca5976542" member_type: VOTER last_known_addr { host: "127.14.147.194" port: 46603 } } peers { permanent_uuid: "43f3886bb1b44e4aabb3e8f71ff6f3b5" member_type: VOTER last_known_addr { host: "127.14.147.193" port: 46859 } } peers { permanent_uuid: "80e8234059154efbbc9417e8b4a71810" member_type: VOTER last_known_addr { host: "127.14.147.195" port: 41003 } }
I20260929 22:52:49.357440 29171 raft_consensus.cc:374] T e4c4c66b006c4d45a1561ab37d0e3cb6 P 43f3886bb1b44e4aabb3e8f71ff6f3b5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 22:52:49.357478 29171 raft_consensus.cc:729] T e4c4c66b006c4d45a1561ab37d0e3cb6 P 43f3886bb1b44e4aabb3e8f71ff6f3b5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 43f3886bb1b44e4aabb3e8f71ff6f3b5, State: Initialized, Role: FOLLOWER
I20260929 22:52:49.357581 29171 consensus_queue.cc:260] T e4c4c66b006c4d45a1561ab37d0e3cb6 P 43f3886bb1b44e4aabb3e8f71ff6f3b5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae4204ae8cb54f4f8f9a085ca5976542" member_type: VOTER last_known_addr { host: "127.14.147.194" port: 46603 } } peers { permanent_uuid: "43f3886bb1b44e4aabb3e8f71ff6f3b5" member_type: VOTER last_known_addr { host: "127.14.147.193" port: 46859 } } peers { permanent_uuid: "80e8234059154efbbc9417e8b4a71810" member_type: VOTER last_known_addr { host: "127.14.147.195" port: 41003 } }
I20260929 22:52:49.357808 29171 ts_tablet_manager.cc:1428] T e4c4c66b006c4d45a1561ab37d0e3cb6 P 43f3886bb1b44e4aabb3e8f71ff6f3b5: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260929 22:52:49.357846 28887 heartbeater.cc:503] Master 127.14.147.254:35819 was elected leader, sending a full tablet report...
W20260929 22:52:49.428789 29022 tablet.cc:2369] T e4c4c66b006c4d45a1561ab37d0e3cb6 P ae4204ae8cb54f4f8f9a085ca5976542: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260929 22:52:49.530721 28888 tablet.cc:2369] T e4c4c66b006c4d45a1561ab37d0e3cb6 P 43f3886bb1b44e4aabb3e8f71ff6f3b5: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260929 22:52:49.572017 29156 tablet.cc:2369] T e4c4c66b006c4d45a1561ab37d0e3cb6 P 80e8234059154efbbc9417e8b4a71810: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 22:52:49.572608 29176 raft_consensus.cc:482] T e4c4c66b006c4d45a1561ab37d0e3cb6 P ae4204ae8cb54f4f8f9a085ca5976542 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 22:52:49.572716 29176 raft_consensus.cc:504] T e4c4c66b006c4d45a1561ab37d0e3cb6 P ae4204ae8cb54f4f8f9a085ca5976542 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae4204ae8cb54f4f8f9a085ca5976542" member_type: VOTER last_known_addr { host: "127.14.147.194" port: 46603 } } peers { permanent_uuid: "43f3886bb1b44e4aabb3e8f71ff6f3b5" member_type: VOTER last_known_addr { host: "127.14.147.193" port: 46859 } } peers { permanent_uuid: "80e8234059154efbbc9417e8b4a71810" member_type: VOTER last_known_addr { host: "127.14.147.195" port: 41003 } }
I20260929 22:52:49.573082 29176 leader_election.cc:288] T e4c4c66b006c4d45a1561ab37d0e3cb6 P ae4204ae8cb54f4f8f9a085ca5976542 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 43f3886bb1b44e4aabb3e8f71ff6f3b5 (127.14.147.193:46859), 80e8234059154efbbc9417e8b4a71810 (127.14.147.195:41003)
I20260929 22:52:49.576148 29165 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:52:49.573344 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.147.194:46841 (local address 127.14.147.193:46859)
0929 22:52:49.573392 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0929 22:52:49.573394 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 22:52:49.573404 (+ 10us) server_negotiation.cc:407] Connection header received
0929 22:52:49.573467 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 22:52:49.573468 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 22:52:49.573478 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 22:52:49.573527 (+ 49us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 22:52:49.573711 (+ 184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:52:49.574401 (+ 690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:52:49.575377 (+ 976us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:52:49.575792 (+ 415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:52:49.575853 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 22:52:49.575860 (+ 7us) server_negotiation.cc:1035] Waiting for connection context
0929 22:52:49.576028 (+ 168us) server_negotiation.cc:299] Negotiation successful
0929 22:52:49.576072 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260929 22:52:49.576159 29168 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:52:49.573441 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.147.194:56083 (local address 127.14.147.195:41003)
0929 22:52:49.573482 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0929 22:52:49.573483 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0929 22:52:49.573582 (+ 99us) server_negotiation.cc:407] Connection header received
0929 22:52:49.573615 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 22:52:49.573616 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 22:52:49.573625 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 22:52:49.573685 (+ 60us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 22:52:49.573843 (+ 158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:52:49.574485 (+ 642us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:52:49.575408 (+ 923us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:52:49.575830 (+ 422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:52:49.575904 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 22:52:49.575913 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0929 22:52:49.576053 (+ 140us) server_negotiation.cc:299] Negotiation successful
0929 22:52:49.576094 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260929 22:52:49.576161 29023 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:52:49.573257 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.147.193:46859 (local address 127.14.147.194:46841)
0929 22:52:49.573317 (+ 60us) negotiation.cc:107] Waiting for socket to connect
0929 22:52:49.573333 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 22:52:49.573402 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 22:52:49.573557 (+ 155us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 22:52:49.573559 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 22:52:49.573561 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 22:52:49.573642 (+ 81us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 22:52:49.573644 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 22:52:49.574479 (+ 835us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 22:52:49.574481 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 22:52:49.575276 (+ 795us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 22:52:49.575279 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 22:52:49.575871 (+ 592us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 22:52:49.575872 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 22:52:49.575955 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 22:52:49.575959 (+ 4us) client_negotiation.cc:769] Sending connection context
0929 22:52:49.576006 (+ 47us) client_negotiation.cc:240] Negotiation successful
0929 22:52:49.576052 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":31,"mutex_wait_us":1}
I20260929 22:52:49.576174 29178 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:52:49.573357 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.147.195:41003 (local address 127.14.147.194:56083)
0929 22:52:49.573517 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0929 22:52:49.573526 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0929 22:52:49.573573 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 22:52:49.573692 (+ 119us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 22:52:49.573693 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 22:52:49.573694 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 22:52:49.573767 (+ 73us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 22:52:49.573768 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 22:52:49.574561 (+ 793us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 22:52:49.574562 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 22:52:49.575309 (+ 747us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 22:52:49.575311 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 22:52:49.575917 (+ 606us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 22:52:49.575918 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 22:52:49.575982 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 22:52:49.575985 (+ 3us) client_negotiation.cc:769] Sending connection context
0929 22:52:49.576039 (+ 54us) client_negotiation.cc:240] Negotiation successful
0929 22:52:49.576077 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":144,"mutex_wait_us":2,"thread_start_us":110,"threads_started":1}
I20260929 22:52:49.576538 28841 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4c4c66b006c4d45a1561ab37d0e3cb6" candidate_uuid: "ae4204ae8cb54f4f8f9a085ca5976542" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "43f3886bb1b44e4aabb3e8f71ff6f3b5" is_pre_election: true
I20260929 22:52:49.576725 28841 raft_consensus.cc:2454] T e4c4c66b006c4d45a1561ab37d0e3cb6 P 43f3886bb1b44e4aabb3e8f71ff6f3b5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae4204ae8cb54f4f8f9a085ca5976542 in term 0.
I20260929 22:52:49.576740 29109 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4c4c66b006c4d45a1561ab37d0e3cb6" candidate_uuid: "ae4204ae8cb54f4f8f9a085ca5976542" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "80e8234059154efbbc9417e8b4a71810" is_pre_election: true
I20260929 22:52:49.576889 29109 raft_consensus.cc:2454] T e4c4c66b006c4d45a1561ab37d0e3cb6 P 80e8234059154efbbc9417e8b4a71810 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae4204ae8cb54f4f8f9a085ca5976542 in term 0.
I20260929 22:52:49.576988 28908 leader_election.cc:302] T e4c4c66b006c4d45a1561ab37d0e3cb6 P ae4204ae8cb54f4f8f9a085ca5976542 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 43f3886bb1b44e4aabb3e8f71ff6f3b5, ae4204ae8cb54f4f8f9a085ca5976542; no voters:
I20260929 22:52:49.577134 29176 raft_consensus.cc:2781] T e4c4c66b006c4d45a1561ab37d0e3cb6 P ae4204ae8cb54f4f8f9a085ca5976542 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 22:52:49.577198 29176 raft_consensus.cc:482] T e4c4c66b006c4d45a1561ab37d0e3cb6 P ae4204ae8cb54f4f8f9a085ca5976542 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 22:52:49.577214 29176 raft_consensus.cc:3037] T e4c4c66b006c4d45a1561ab37d0e3cb6 P ae4204ae8cb54f4f8f9a085ca5976542 [term 0 FOLLOWER]: Advancing to term 1
I20260929 22:52:49.577929 29176 raft_consensus.cc:504] T e4c4c66b006c4d45a1561ab37d0e3cb6 P ae4204ae8cb54f4f8f9a085ca5976542 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae4204ae8cb54f4f8f9a085ca5976542" member_type: VOTER last_known_addr { host: "127.14.147.194" port: 46603 } } peers { permanent_uuid: "43f3886bb1b44e4aabb3e8f71ff6f3b5" member_type: VOTER last_known_addr { host: "127.14.147.193" port: 46859 } } peers { permanent_uuid: "80e8234059154efbbc9417e8b4a71810" member_type: VOTER last_known_addr { host: "127.14.147.195" port: 41003 } }
I20260929 22:52:49.578047 29176 leader_election.cc:288] T e4c4c66b006c4d45a1561ab37d0e3cb6 P ae4204ae8cb54f4f8f9a085ca5976542 [CANDIDATE]: Term 1 election: Requested vote from peers 43f3886bb1b44e4aabb3e8f71ff6f3b5 (127.14.147.193:46859), 80e8234059154efbbc9417e8b4a71810 (127.14.147.195:41003)
I20260929 22:52:49.578249 28841 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4c4c66b006c4d45a1561ab37d0e3cb6" candidate_uuid: "ae4204ae8cb54f4f8f9a085ca5976542" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "43f3886bb1b44e4aabb3e8f71ff6f3b5"
I20260929 22:52:49.578261 29109 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4c4c66b006c4d45a1561ab37d0e3cb6" candidate_uuid: "ae4204ae8cb54f4f8f9a085ca5976542" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "80e8234059154efbbc9417e8b4a71810"
I20260929 22:52:49.578329 28841 raft_consensus.cc:3037] T e4c4c66b006c4d45a1561ab37d0e3cb6 P 43f3886bb1b44e4aabb3e8f71ff6f3b5 [term 0 FOLLOWER]: Advancing to term 1
I20260929 22:52:49.578344 29109 raft_consensus.cc:3037] T e4c4c66b006c4d45a1561ab37d0e3cb6 P 80e8234059154efbbc9417e8b4a71810 [term 0 FOLLOWER]: Advancing to term 1
I20260929 22:52:49.579046 28841 raft_consensus.cc:2454] T e4c4c66b006c4d45a1561ab37d0e3cb6 P 43f3886bb1b44e4aabb3e8f71ff6f3b5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ae4204ae8cb54f4f8f9a085ca5976542 in term 1.
I20260929 22:52:49.579062 29109 raft_consensus.cc:2454] T e4c4c66b006c4d45a1561ab37d0e3cb6 P 80e8234059154efbbc9417e8b4a71810 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ae4204ae8cb54f4f8f9a085ca5976542 in term 1.
I20260929 22:52:49.579203 28908 leader_election.cc:302] T e4c4c66b006c4d45a1561ab37d0e3cb6 P ae4204ae8cb54f4f8f9a085ca5976542 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 43f3886bb1b44e4aabb3e8f71ff6f3b5, ae4204ae8cb54f4f8f9a085ca5976542; no voters:
I20260929 22:52:49.579308 29176 raft_consensus.cc:2781] T e4c4c66b006c4d45a1561ab37d0e3cb6 P ae4204ae8cb54f4f8f9a085ca5976542 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 22:52:49.579471 29176 raft_consensus.cc:686] T e4c4c66b006c4d45a1561ab37d0e3cb6 P ae4204ae8cb54f4f8f9a085ca5976542 [term 1 LEADER]: Becoming Leader. State: Replica: ae4204ae8cb54f4f8f9a085ca5976542, State: Running, Role: LEADER
I20260929 22:52:49.579562 29176 consensus_queue.cc:237] T e4c4c66b006c4d45a1561ab37d0e3cb6 P ae4204ae8cb54f4f8f9a085ca5976542 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae4204ae8cb54f4f8f9a085ca5976542" member_type: VOTER last_known_addr { host: "127.14.147.194" port: 46603 } } peers { permanent_uuid: "43f3886bb1b44e4aabb3e8f71ff6f3b5" member_type: VOTER last_known_addr { host: "127.14.147.193" port: 46859 } } peers { permanent_uuid: "80e8234059154efbbc9417e8b4a71810" member_type: VOTER last_known_addr { host: "127.14.147.195" port: 41003 } }
I20260929 22:52:49.580384 28696 catalog_manager.cc:5495] T e4c4c66b006c4d45a1561ab37d0e3cb6 P ae4204ae8cb54f4f8f9a085ca5976542 reported cstate change: term changed from 0 to 1, leader changed from <none> to ae4204ae8cb54f4f8f9a085ca5976542 (127.14.147.194). New cstate: current_term: 1 leader_uuid: "ae4204ae8cb54f4f8f9a085ca5976542" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae4204ae8cb54f4f8f9a085ca5976542" member_type: VOTER last_known_addr { host: "127.14.147.194" port: 46603 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "43f3886bb1b44e4aabb3e8f71ff6f3b5" member_type: VOTER last_known_addr { host: "127.14.147.193" port: 46859 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "80e8234059154efbbc9417e8b4a71810" member_type: VOTER last_known_addr { host: "127.14.147.195" port: 41003 } health_report { overall_health: UNKNOWN } } }
I20260929 22:52:49.634748 29166 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:52:49.632309 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53946 (local address 127.14.147.194:46603)
0929 22:52:49.632363 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0929 22:52:49.632364 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0929 22:52:49.632371 (+ 7us) server_negotiation.cc:407] Connection header received
0929 22:52:49.632416 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 22:52:49.632416 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 22:52:49.632434 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 22:52:49.632504 (+ 70us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 22:52:49.632698 (+ 194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:52:49.633241 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:52:49.634051 (+ 810us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:52:49.634259 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:52:49.634330 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 22:52:49.634451 (+ 121us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 22:52:49.634555 (+ 104us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 22:52:49.634619 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0929 22:52:49.634636 (+ 17us) server_negotiation.cc:299] Negotiation successful
0929 22:52:49.634655 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260929 22:52:49.636528 28841 raft_consensus.cc:1261] T e4c4c66b006c4d45a1561ab37d0e3cb6 P 43f3886bb1b44e4aabb3e8f71ff6f3b5 [term 1 FOLLOWER]: Refusing update from remote peer ae4204ae8cb54f4f8f9a085ca5976542: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 22:52:49.636528 29109 raft_consensus.cc:1261] T e4c4c66b006c4d45a1561ab37d0e3cb6 P 80e8234059154efbbc9417e8b4a71810 [term 1 FOLLOWER]: Refusing update from remote peer ae4204ae8cb54f4f8f9a085ca5976542: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 22:52:49.636849 29176 consensus_queue.cc:1035] T e4c4c66b006c4d45a1561ab37d0e3cb6 P ae4204ae8cb54f4f8f9a085ca5976542 [LEADER]: Connected to new peer: Peer: permanent_uuid: "80e8234059154efbbc9417e8b4a71810" member_type: VOTER last_known_addr { host: "127.14.147.195" port: 41003 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 22:52:49.637027 29179 consensus_queue.cc:1035] T e4c4c66b006c4d45a1561ab37d0e3cb6 P ae4204ae8cb54f4f8f9a085ca5976542 [LEADER]: Connected to new peer: Peer: permanent_uuid: "43f3886bb1b44e4aabb3e8f71ff6f3b5" member_type: VOTER last_known_addr { host: "127.14.147.193" port: 46859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 22:52:49.639328 29182 mvcc.cc:205] Tried to move back new op lower bound from 7334798826028654592 to 7334798825798463488. Current Snapshot: MvccSnapshot[applied={T|T < 7334798826028654592}]
I20260929 22:52:49.639475 29184 mvcc.cc:205] Tried to move back new op lower bound from 7334798826028654592 to 7334798825798463488. Current Snapshot: MvccSnapshot[applied={T|T < 7334798826028654592}]
I20260929 22:52:49.641664 28695 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54110:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 22:52:49.641770 28695 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54110:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 22:52:49.642540 28695 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 83dd29106e1946398652603810176095: Sending DeleteTablet for 3 replicas of tablet e4c4c66b006c4d45a1561ab37d0e3cb6
I20260929 22:52:49.642838 28955 tablet_service.cc:1514] Processing DeleteTablet for tablet e4c4c66b006c4d45a1561ab37d0e3cb6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 22:52:49 UTC) from {username='kudu', principal='kudu/127.14.147.254@KRBTEST.COM'} at 127.0.0.1:53942
I20260929 22:52:49.642910 14927 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhWOCLk/build/release/bin/kudu with pid 28758
I20260929 22:52:49.642913 28821 tablet_service.cc:1514] Processing DeleteTablet for tablet e4c4c66b006c4d45a1561ab37d0e3cb6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 22:52:49 UTC) from {username='kudu', principal='kudu/127.14.147.254@KRBTEST.COM'} at 127.0.0.1:58728
I20260929 22:52:49.643818 29192 tablet_replica.cc:330] T e4c4c66b006c4d45a1561ab37d0e3cb6 P ae4204ae8cb54f4f8f9a085ca5976542: stopping tablet replica
I20260929 22:52:49.643808 29089 tablet_service.cc:1514] Processing DeleteTablet for tablet e4c4c66b006c4d45a1561ab37d0e3cb6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 22:52:49 UTC) from {username='kudu', principal='kudu/127.14.147.254@KRBTEST.COM'} at 127.0.0.1:34162
I20260929 22:52:49.643926 29192 raft_consensus.cc:2229] T e4c4c66b006c4d45a1561ab37d0e3cb6 P ae4204ae8cb54f4f8f9a085ca5976542 [term 1 LEADER]: Raft consensus shutting down.
I20260929 22:52:49.644052 29192 raft_consensus.cc:2258] T e4c4c66b006c4d45a1561ab37d0e3cb6 P ae4204ae8cb54f4f8f9a085ca5976542 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 22:52:49.644201 29193 tablet_replica.cc:330] T e4c4c66b006c4d45a1561ab37d0e3cb6 P 80e8234059154efbbc9417e8b4a71810: stopping tablet replica
I20260929 22:52:49.644304 29193 raft_consensus.cc:2229] T e4c4c66b006c4d45a1561ab37d0e3cb6 P 80e8234059154efbbc9417e8b4a71810 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 22:52:49.644408 29193 raft_consensus.cc:2258] T e4c4c66b006c4d45a1561ab37d0e3cb6 P 80e8234059154efbbc9417e8b4a71810 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 22:52:49.644536 29192 ts_tablet_manager.cc:1905] T e4c4c66b006c4d45a1561ab37d0e3cb6 P ae4204ae8cb54f4f8f9a085ca5976542: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 22:52:49.644855 29193 ts_tablet_manager.cc:1905] T e4c4c66b006c4d45a1561ab37d0e3cb6 P 80e8234059154efbbc9417e8b4a71810: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 22:52:49.646407 29193 ts_tablet_manager.cc:1918] T e4c4c66b006c4d45a1561ab37d0e3cb6 P 80e8234059154efbbc9417e8b4a71810: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 22:52:49.646430 29192 ts_tablet_manager.cc:1918] T e4c4c66b006c4d45a1561ab37d0e3cb6 P ae4204ae8cb54f4f8f9a085ca5976542: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 22:52:49.646489 29192 log.cc:1196] T e4c4c66b006c4d45a1561ab37d0e3cb6 P ae4204ae8cb54f4f8f9a085ca5976542: Deleting WAL directory at /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790722213368738-14927-0/minicluster-data/ts-1/wal/wals/e4c4c66b006c4d45a1561ab37d0e3cb6
I20260929 22:52:49.646489 29193 log.cc:1196] T e4c4c66b006c4d45a1561ab37d0e3cb6 P 80e8234059154efbbc9417e8b4a71810: Deleting WAL directory at /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790722213368738-14927-0/minicluster-data/ts-2/wal/wals/e4c4c66b006c4d45a1561ab37d0e3cb6
I20260929 22:52:49.646768 29192 ts_tablet_manager.cc:1939] T e4c4c66b006c4d45a1561ab37d0e3cb6 P ae4204ae8cb54f4f8f9a085ca5976542: Deleting consensus metadata
I20260929 22:52:49.646791 29193 ts_tablet_manager.cc:1939] T e4c4c66b006c4d45a1561ab37d0e3cb6 P 80e8234059154efbbc9417e8b4a71810: Deleting consensus metadata
I20260929 22:52:49.647219 28680 catalog_manager.cc:4841] TS 80e8234059154efbbc9417e8b4a71810 (127.14.147.195:41003): tablet e4c4c66b006c4d45a1561ab37d0e3cb6 (table test-table [id=58881d3f259544f09bb6295e9259719f]) successfully deleted
I20260929 22:52:49.647234 28681 catalog_manager.cc:4841] TS ae4204ae8cb54f4f8f9a085ca5976542 (127.14.147.194:46603): tablet e4c4c66b006c4d45a1561ab37d0e3cb6 (table test-table [id=58881d3f259544f09bb6295e9259719f]) successfully deleted
W20260929 22:52:49.647742 28680 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.14.147.193:46859 (error 108)
I20260929 22:52:49.648247 14927 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhWOCLk/build/release/bin/kudu with pid 28892
I20260929 22:52:49.648424 29163 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:52:49.648275 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.147.193:46859 (local address 127.0.0.1:58740)
0929 22:52:49.648336 (+ 61us) negotiation.cc:107] Waiting for socket to connect
0929 22:52:49.648389 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.147.193:46859: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
W20260929 22:52:49.648845 28680 catalog_manager.cc:4584] TS 43f3886bb1b44e4aabb3e8f71ff6f3b5 (127.14.147.193:46859): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet e4c4c66b006c4d45a1561ab37d0e3cb6: Network error: Client connection negotiation failed: client connection to 127.14.147.193:46859: connect: Connection refused (error 111)
I20260929 22:52:49.653678 14927 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhWOCLk/build/release/bin/kudu with pid 29026
I20260929 22:52:49.657886 14927 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhWOCLk/build/release/bin/kudu with pid 28664
I20260929 22:52:49.662609 14927 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhWOCLk/build/release/bin/kudu
/tmp/dist-test-taskhWOCLk/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790722213368738-14927-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790722213368738-14927-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790722213368738-14927-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790722213368738-14927-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.147.254:35819
--webserver_interface=127.14.147.254
--webserver_port=33785
--keytab_file=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790722213368738-14927-0/krb5kdc/kudu_127.14.147.254.keytab
--principal=kudu/127.14.147.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.147.212:43149
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.147.254:35819
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--ipki_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790722213368738-14927-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790722213368738-14927-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790722213368738-14927-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 22:52:49.741815 29195 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 22:52:49.742044 29195 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 22:52:49.742067 29195 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 22:52:49.743294 29195 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 22:52:49.743346 29195 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 22:52:49.743359 29195 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 22:52:49.743371 29195 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 22:52:49.743389 29195 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 22:52:49.745008 29195 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.147.212:43149
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790722213368738-14927-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790722213368738-14927-0/minicluster-data/master-0/wal
--ipki_private_key_password_cmd=echo foo
--ipki_ca_key_size=768
--master_addresses=127.14.147.254:35819
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790722213368738-14927-0/krb5kdc/kudu_127.14.147.254.keytab
--principal=kudu/127.14.147.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.147.254:35819
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790722213368738-14927-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.14.147.254
--webserver_port=33785
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29195
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790722213368738-14927-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 21:58:54 UTC on ccycloud.quasar-xguoti.root.comops.site
build id 4681390
I20260929 22:52:49.745275 29195 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 22:52:49.745500 29195 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 22:52:49.755311 29204 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 22:52:49.755445 29195 server_base.cc:1034] running on GCE node
W20260929 22:52:49.755290 29202 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 22:52:49.755282 29201 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 22:52:49.755743 29195 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 22:52:49.756070 29195 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 22:52:49.757208 29195 hybrid_clock.cc:648] HybridClock initialized: now 1790722369757203 us; error 32 us; skew 500 ppm
Sep 29 22:52:49 dist-test-slave-98mk krb5kdc[28651](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790722369, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.147.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 22:52:49.760048 29195 init.cc:377] Logged in from keytab as kudu/127.14.147.254@KRBTEST.COM (short username kudu)
I20260929 22:52:49.760799 29195 webserver.cc:460] Webserver started at http://127.14.147.254:33785/ using document root <none> and password file <none>
I20260929 22:52:49.761103 29195 fs_manager.cc:359] Metadata directory not provided
I20260929 22:52:49.761180 29195 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 22:52:49.762722 29195 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.002s
I20260929 22:52:49.763361 29211 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 22:52:49.763547 29195 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260929 22:52:49.763636 29195 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790722213368738-14927-0/minicluster-data/master-0/data,/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790722213368738-14927-0/minicluster-data/master-0/wal
uuid: "83dd29106e1946398652603810176095"
format_stamp: "Formatted at 2026-09-29 22:52:48 on dist-test-slave-98mk"
server_key: "dfa6a553431dbd9d21657950be5ff375"
server_key_iv: "dee84b8c1d392b6d5fef2d8ffe935993"
server_key_version: "encryptionkey@0"
I20260929 22:52:49.764110 29195 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790722213368738-14927-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790722213368738-14927-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790722213368738-14927-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 22:52:49.782848 29195 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 22:52:49.785109 29195 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 22:52:49.785255 29195 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 22:52:49.789767 29195 rpc_server.cc:307] RPC server started. Bound to: 127.14.147.254:35819
I20260929 22:52:49.789894 29263 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.147.254:35819 every 8 connection(s)
I20260929 22:52:49.790223 29195 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790722213368738-14927-0/minicluster-data/master-0/data/info.pb
I20260929 22:52:49.791970 29264 sys_catalog.cc:263] Verifying existing consensus state
I20260929 22:52:49.792667 29264 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 83dd29106e1946398652603810176095: Bootstrap starting.
I20260929 22:52:49.797703 14927 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhWOCLk/build/release/bin/kudu as pid 29195
I20260929 22:52:49.798000 14927 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhWOCLk/build/release/bin/kudu
/tmp/dist-test-taskhWOCLk/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790722213368738-14927-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790722213368738-14927-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790722213368738-14927-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790722213368738-14927-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.147.193:46859
--local_ip_for_outbound_sockets=127.14.147.193
--tserver_master_addrs=127.14.147.254:35819
--webserver_port=35563
--webserver_interface=127.14.147.193
--keytab_file=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790722213368738-14927-0/krb5kdc/kudu_127.14.147.193.keytab
--principal=kudu/127.14.147.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.147.212:43149
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790722213368738-14927-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790722213368738-14927-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790722213368738-14927-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 22:52:49.801362 29264 log.cc:824] T 00000000000000000000000000000000 P 83dd29106e1946398652603810176095: Log is configured to *not* fsync() on all Append() calls
I20260929 22:52:49.804131 29264 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 83dd29106e1946398652603810176095: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260929 22:52:49.804509 29264 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 83dd29106e1946398652603810176095: Bootstrap complete.
I20260929 22:52:49.807924 29264 raft_consensus.cc:348] T 00000000000000000000000000000000 P 83dd29106e1946398652603810176095 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "83dd29106e1946398652603810176095" member_type: VOTER last_known_addr { host: "127.14.147.254" port: 35819 } }
I20260929 22:52:49.808377 29264 raft_consensus.cc:729] T 00000000000000000000000000000000 P 83dd29106e1946398652603810176095 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 83dd29106e1946398652603810176095, State: Initialized, Role: FOLLOWER
I20260929 22:52:49.808629 29264 consensus_queue.cc:260] T 00000000000000000000000000000000 P 83dd29106e1946398652603810176095 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "83dd29106e1946398652603810176095" member_type: VOTER last_known_addr { host: "127.14.147.254" port: 35819 } }
I20260929 22:52:49.808738 29264 raft_consensus.cc:388] T 00000000000000000000000000000000 P 83dd29106e1946398652603810176095 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 22:52:49.808825 29264 raft_consensus.cc:482] T 00000000000000000000000000000000 P 83dd29106e1946398652603810176095 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 22:52:49.808890 29264 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 83dd29106e1946398652603810176095 [term 1 FOLLOWER]: Advancing to term 2
I20260929 22:52:49.810396 29264 raft_consensus.cc:504] T 00000000000000000000000000000000 P 83dd29106e1946398652603810176095 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "83dd29106e1946398652603810176095" member_type: VOTER last_known_addr { host: "127.14.147.254" port: 35819 } }
I20260929 22:52:49.810516 29264 leader_election.cc:302] T 00000000000000000000000000000000 P 83dd29106e1946398652603810176095 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 83dd29106e1946398652603810176095; no voters:
I20260929 22:52:49.810717 29264 leader_election.cc:288] T 00000000000000000000000000000000 P 83dd29106e1946398652603810176095 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260929 22:52:49.810779 29268 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 83dd29106e1946398652603810176095 [term 2 FOLLOWER]: Leader election won for term 2
I20260929 22:52:49.810950 29268 raft_consensus.cc:686] T 00000000000000000000000000000000 P 83dd29106e1946398652603810176095 [term 2 LEADER]: Becoming Leader. State: Replica: 83dd29106e1946398652603810176095, State: Running, Role: LEADER
I20260929 22:52:49.810976 29264 sys_catalog.cc:564] T 00000000000000000000000000000000 P 83dd29106e1946398652603810176095 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 22:52:49.811065 29268 consensus_queue.cc:237] T 00000000000000000000000000000000 P 83dd29106e1946398652603810176095 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "83dd29106e1946398652603810176095" member_type: VOTER last_known_addr { host: "127.14.147.254" port: 35819 } }
I20260929 22:52:49.811344 29269 sys_catalog.cc:455] T 00000000000000000000000000000000 P 83dd29106e1946398652603810176095 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "83dd29106e1946398652603810176095" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "83dd29106e1946398652603810176095" member_type: VOTER last_known_addr { host: "127.14.147.254" port: 35819 } } }
I20260929 22:52:49.811355 29270 sys_catalog.cc:455] T 00000000000000000000000000000000 P 83dd29106e1946398652603810176095 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 83dd29106e1946398652603810176095. Latest consensus state: current_term: 2 leader_uuid: "83dd29106e1946398652603810176095" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "83dd29106e1946398652603810176095" member_type: VOTER last_known_addr { host: "127.14.147.254" port: 35819 } } }
I20260929 22:52:49.811506 29270 sys_catalog.cc:458] T 00000000000000000000000000000000 P 83dd29106e1946398652603810176095 [sys.catalog]: This master's current role is: LEADER
I20260929 22:52:49.811465 29269 sys_catalog.cc:458] T 00000000000000000000000000000000 P 83dd29106e1946398652603810176095 [sys.catalog]: This master's current role is: LEADER
I20260929 22:52:49.811751 29276 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 22:52:49.812582 29276 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 22:52:49.812691 29276 catalog_manager.cc:1259] Loaded cluster ID: 56089a3d64ff483fa50c5dd79b352240
I20260929 22:52:49.812738 29276 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 22:52:49.813221 29276 catalog_manager.cc:1314] IPKI private key was stored in cleartext, attempting to encrypt it.
I20260929 22:52:49.817783 29276 catalog_manager.cc:1324] IPKI private key successfully encrypted.
I20260929 22:52:49.820526 29276 catalog_manager.cc:1504] Loading token signing keys...
I20260929 22:52:49.820770 29276 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 83dd29106e1946398652603810176095: Loaded TSK: 0
I20260929 22:52:49.821008 29276 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260929 22:52:49.883855 29266 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 22:52:49.884071 29266 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 22:52:49.884104 29266 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 22:52:49.885309 29266 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 22:52:49.885360 29266 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 22:52:49.885398 29266 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.147.193
I20260929 22:52:49.887053 29266 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.147.212:43149
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790722213368738-14927-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790722213368738-14927-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790722213368738-14927-0/krb5kdc/kudu_127.14.147.193.keytab
--principal=kudu/127.14.147.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.147.193:46859
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790722213368738-14927-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.14.147.193
--webserver_port=35563
--tserver_master_addrs=127.14.147.254:35819
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29266
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.147.193
--log_dir=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790722213368738-14927-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 21:58:54 UTC on ccycloud.quasar-xguoti.root.comops.site
build id 4681390
I20260929 22:52:49.887504 29266 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 22:52:49.887940 29266 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 22:52:49.893191 29294 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 22:52:49.893206 29291 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 22:52:49.893420 29266 server_base.cc:1034] running on GCE node
W20260929 22:52:49.893216 29292 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 22:52:49.893713 29266 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 22:52:49.893976 29266 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 22:52:49.895148 29266 hybrid_clock.cc:648] HybridClock initialized: now 1790722369895121 us; error 42 us; skew 500 ppm
Sep 29 22:52:49 dist-test-slave-98mk krb5kdc[28651](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790722369, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.147.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 22:52:49.903107 29266 init.cc:377] Logged in from keytab as kudu/127.14.147.193@KRBTEST.COM (short username kudu)
I20260929 22:52:49.903671 29266 webserver.cc:460] Webserver started at http://127.14.147.193:35563/ using document root <none> and password file <none>
I20260929 22:52:49.903900 29266 fs_manager.cc:359] Metadata directory not provided
I20260929 22:52:49.903966 29266 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 22:52:49.905954 29266 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.002s sys 0.000s
I20260929 22:52:49.906728 29301 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 22:52:49.906901 29266 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260929 22:52:49.906994 29266 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790722213368738-14927-0/minicluster-data/ts-0/data,/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790722213368738-14927-0/minicluster-data/ts-0/wal
uuid: "43f3886bb1b44e4aabb3e8f71ff6f3b5"
format_stamp: "Formatted at 2026-09-29 22:52:49 on dist-test-slave-98mk"
server_key: "85d4f73702f5e7e081d2846fbdda5227"
server_key_iv: "933fc3cece9f429af97ebdb9a876d23e"
server_key_version: "encryptionkey@0"
I20260929 22:52:49.907349 29266 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790722213368738-14927-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790722213368738-14927-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790722213368738-14927-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 22:52:49.922989 29266 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 22:52:49.925179 29266 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 22:52:49.925323 29266 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 22:52:49.925554 29266 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 22:52:49.926003 29308 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260929 22:52:49.927433 29266 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260929 22:52:49.927489 29266 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260929 22:52:49.927521 29266 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260929 22:52:49.928161 29266 ts_tablet_manager.cc:610] Registered 1 tablets
I20260929 22:52:49.928202 29266 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20260929 22:52:49.928267 29308 tablet_bootstrap.cc:492] T e4c4c66b006c4d45a1561ab37d0e3cb6 P 43f3886bb1b44e4aabb3e8f71ff6f3b5: Bootstrap starting.
I20260929 22:52:49.935885 29266 rpc_server.cc:307] RPC server started. Bound to: 127.14.147.193:46859
I20260929 22:52:49.936398 29266 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790722213368738-14927-0/minicluster-data/ts-0/data/info.pb
I20260929 22:52:49.939193 29308 log.cc:824] T e4c4c66b006c4d45a1561ab37d0e3cb6 P 43f3886bb1b44e4aabb3e8f71ff6f3b5: Log is configured to *not* fsync() on all Append() calls
I20260929 22:52:49.939492 29415 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.147.193:46859 every 8 connection(s)
I20260929 22:52:49.944157 14927 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhWOCLk/build/release/bin/kudu as pid 29266
I20260929 22:52:49.944463 14927 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhWOCLk/build/release/bin/kudu
/tmp/dist-test-taskhWOCLk/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790722213368738-14927-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790722213368738-14927-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790722213368738-14927-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790722213368738-14927-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.147.194:46603
--local_ip_for_outbound_sockets=127.14.147.194
--tserver_master_addrs=127.14.147.254:35819
--webserver_port=41865
--webserver_interface=127.14.147.194
--keytab_file=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790722213368738-14927-0/krb5kdc/kudu_127.14.147.194.keytab
--principal=kudu/127.14.147.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.147.212:43149
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790722213368738-14927-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790722213368738-14927-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790722213368738-14927-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 22:52:49.944500 29308 tablet_bootstrap.cc:492] T e4c4c66b006c4d45a1561ab37d0e3cb6 P 43f3886bb1b44e4aabb3e8f71ff6f3b5: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260929 22:52:49.944796 29308 tablet_bootstrap.cc:492] T e4c4c66b006c4d45a1561ab37d0e3cb6 P 43f3886bb1b44e4aabb3e8f71ff6f3b5: Bootstrap complete.
Sep 29 22:52:49 dist-test-slave-98mk krb5kdc[28651](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790722369, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.147.193@KRBTEST.COM for kudu/127.14.147.254@KRBTEST.COM
I20260929 22:52:49.945914 29308 ts_tablet_manager.cc:1397] T e4c4c66b006c4d45a1561ab37d0e3cb6 P 43f3886bb1b44e4aabb3e8f71ff6f3b5: Time spent bootstrapping tablet: real 0.018s user 0.007s sys 0.006s
I20260929 22:52:49.947242 29308 raft_consensus.cc:348] T e4c4c66b006c4d45a1561ab37d0e3cb6 P 43f3886bb1b44e4aabb3e8f71ff6f3b5 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae4204ae8cb54f4f8f9a085ca5976542" member_type: VOTER last_known_addr { host: "127.14.147.194" port: 46603 } } peers { permanent_uuid: "43f3886bb1b44e4aabb3e8f71ff6f3b5" member_type: VOTER last_known_addr { host: "127.14.147.193" port: 46859 } } peers { permanent_uuid: "80e8234059154efbbc9417e8b4a71810" member_type: VOTER last_known_addr { host: "127.14.147.195" port: 41003 } }
I20260929 22:52:49.947626 29308 raft_consensus.cc:729] T e4c4c66b006c4d45a1561ab37d0e3cb6 P 43f3886bb1b44e4aabb3e8f71ff6f3b5 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 43f3886bb1b44e4aabb3e8f71ff6f3b5, State: Initialized, Role: FOLLOWER
I20260929 22:52:49.947796 29308 consensus_queue.cc:260] T e4c4c66b006c4d45a1561ab37d0e3cb6 P 43f3886bb1b44e4aabb3e8f71ff6f3b5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae4204ae8cb54f4f8f9a085ca5976542" member_type: VOTER last_known_addr { host: "127.14.147.194" port: 46603 } } peers { permanent_uuid: "43f3886bb1b44e4aabb3e8f71ff6f3b5" member_type: VOTER last_known_addr { host: "127.14.147.193" port: 46859 } } peers { permanent_uuid: "80e8234059154efbbc9417e8b4a71810" member_type: VOTER last_known_addr { host: "127.14.147.195" port: 41003 } }
I20260929 22:52:49.948098 29308 ts_tablet_manager.cc:1428] T e4c4c66b006c4d45a1561ab37d0e3cb6 P 43f3886bb1b44e4aabb3e8f71ff6f3b5: Time spent starting tablet: real 0.002s user 0.004s sys 0.000s
I20260929 22:52:49.949896 29419 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:52:49.937303 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.147.193:47407 (local address 127.14.147.254:35819)
0929 22:52:49.937483 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0929 22:52:49.937493 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 22:52:49.937760 (+ 267us) server_negotiation.cc:407] Connection header received
0929 22:52:49.938491 (+ 731us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 22:52:49.938502 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 22:52:49.938569 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 22:52:49.938740 (+ 171us) server_negotiation.cc:226] Negotiated authn=SASL
0929 22:52:49.939623 (+ 883us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:52:49.941407 (+ 1784us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:52:49.942950 (+ 1543us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:52:49.943582 (+ 632us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:52:49.943731 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 22:52:49.946227 (+ 2496us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 22:52:49.946231 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 22:52:49.946232 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 22:52:49.946244 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0929 22:52:49.948241 (+ 1997us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 22:52:49.948644 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 22:52:49.948648 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 22:52:49.948648 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 22:52:49.948749 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 22:52:49.948974 (+ 225us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 22:52:49.948975 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 22:52:49.948975 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 22:52:49.949214 (+ 239us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 22:52:49.949290 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0929 22:52:49.949571 (+ 281us) server_negotiation.cc:299] Negotiation successful
0929 22:52:49.949696 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":138,"thread_start_us":78,"threads_started":1}
I20260929 22:52:49.949896 29418 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:52:49.937251 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.147.254:35819 (local address 127.14.147.193:47407)
0929 22:52:49.937570 (+ 319us) negotiation.cc:107] Waiting for socket to connect
0929 22:52:49.937613 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0929 22:52:49.938237 (+ 624us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 22:52:49.938761 (+ 524us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 22:52:49.938765 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 22:52:49.939134 (+ 369us) client_negotiation.cc:189] Negotiated authn=SASL
0929 22:52:49.939389 (+ 255us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 22:52:49.939396 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 22:52:49.941633 (+ 2237us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 22:52:49.941635 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 22:52:49.942818 (+ 1183us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 22:52:49.942823 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 22:52:49.943694 (+ 871us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 22:52:49.943696 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 22:52:49.943860 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 22:52:49.944480 (+ 620us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 22:52:49.944488 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0929 22:52:49.946106 (+ 1618us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 22:52:49.948327 (+ 2221us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 22:52:49.948332 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 22:52:49.948334 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 22:52:49.948556 (+ 222us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 22:52:49.948812 (+ 256us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 22:52:49.948813 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 22:52:49.948813 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 22:52:49.948917 (+ 104us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 22:52:49.949281 (+ 364us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 22:52:49.949288 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 22:52:49.949421 (+ 133us) client_negotiation.cc:769] Sending connection context
0929 22:52:49.949508 (+ 87us) client_negotiation.cc:240] Negotiation successful
0929 22:52:49.949692 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":274,"thread_start_us":102,"threads_started":1}
I20260929 22:52:49.950588 29416 heartbeater.cc:347] Connected to a master server at 127.14.147.254:35819
I20260929 22:52:49.950716 29416 heartbeater.cc:464] Registering TS with master...
I20260929 22:52:49.950924 29416 heartbeater.cc:511] Master 127.14.147.254:35819 requested a full tablet report, sending...
I20260929 22:52:49.951654 29228 ts_manager.cc:194] Registered new tserver with Master: 43f3886bb1b44e4aabb3e8f71ff6f3b5 (127.14.147.193:46859)
I20260929 22:52:49.952708 29228 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.147.193@KRBTEST.COM'} at 127.14.147.193:47407
I20260929 22:52:49.953719 29416 heartbeater.cc:503] Master 127.14.147.254:35819 was elected leader, sending a full tablet report...
I20260929 22:52:49.957296 29424 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:52:49.952982 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58754 (local address 127.14.147.193:46859)
0929 22:52:49.953890 (+ 908us) server_negotiation.cc:206] Beginning negotiation
0929 22:52:49.953895 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 22:52:49.953907 (+ 12us) server_negotiation.cc:407] Connection header received
0929 22:52:49.953944 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 22:52:49.953946 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 22:52:49.953961 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 22:52:49.954052 (+ 91us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 22:52:49.954224 (+ 172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:52:49.954884 (+ 660us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:52:49.956465 (+ 1581us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:52:49.956942 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:52:49.957009 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 22:52:49.957046 (+ 37us) server_negotiation.cc:1035] Waiting for connection context
0929 22:52:49.957205 (+ 159us) server_negotiation.cc:299] Negotiation successful
0929 22:52:49.957239 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":873,"thread_start_us":96,"threads_started":1}
I20260929 22:52:49.957334 29423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:52:49.952272 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.147.193:46859 (local address 127.0.0.1:58754)
0929 22:52:49.952448 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0929 22:52:49.952483 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0929 22:52:49.952550 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 22:52:49.954063 (+ 1513us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 22:52:49.954067 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 22:52:49.954069 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 22:52:49.954141 (+ 72us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 22:52:49.954143 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 22:52:49.954968 (+ 825us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 22:52:49.954969 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 22:52:49.956332 (+ 1363us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 22:52:49.956335 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 22:52:49.957019 (+ 684us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 22:52:49.957019 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 22:52:49.957111 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 22:52:49.957126 (+ 15us) client_negotiation.cc:769] Sending connection context
0929 22:52:49.957192 (+ 66us) client_negotiation.cc:240] Negotiation successful
0929 22:52:49.957253 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":144,"thread_start_us":104,"threads_started":1}
I20260929 22:52:49.957759 29350 tablet_service.cc:1514] Processing DeleteTablet for tablet e4c4c66b006c4d45a1561ab37d0e3cb6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 22:52:49 UTC) from {username='kudu', principal='kudu/127.14.147.254@KRBTEST.COM'} at 127.0.0.1:58754
I20260929 22:52:49.958397 29425 tablet_replica.cc:330] stopping tablet replica
I20260929 22:52:49.958508 29425 raft_consensus.cc:2229] T e4c4c66b006c4d45a1561ab37d0e3cb6 P 43f3886bb1b44e4aabb3e8f71ff6f3b5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 22:52:49.958577 29425 raft_consensus.cc:2258] T e4c4c66b006c4d45a1561ab37d0e3cb6 P 43f3886bb1b44e4aabb3e8f71ff6f3b5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 22:52:49.958884 29425 ts_tablet_manager.cc:1905] T e4c4c66b006c4d45a1561ab37d0e3cb6 P 43f3886bb1b44e4aabb3e8f71ff6f3b5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 22:52:49.961016 29425 ts_tablet_manager.cc:1918] T e4c4c66b006c4d45a1561ab37d0e3cb6 P 43f3886bb1b44e4aabb3e8f71ff6f3b5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 22:52:49.961097 29425 log.cc:1196] T e4c4c66b006c4d45a1561ab37d0e3cb6 P 43f3886bb1b44e4aabb3e8f71ff6f3b5: Deleting WAL directory at /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790722213368738-14927-0/minicluster-data/ts-0/wal/wals/e4c4c66b006c4d45a1561ab37d0e3cb6
I20260929 22:52:49.961282 29425 ts_tablet_manager.cc:1939] T e4c4c66b006c4d45a1561ab37d0e3cb6 P 43f3886bb1b44e4aabb3e8f71ff6f3b5: Deleting consensus metadata
I20260929 22:52:49.961843 29212 catalog_manager.cc:4841] TS 43f3886bb1b44e4aabb3e8f71ff6f3b5 (127.14.147.193:46859): tablet e4c4c66b006c4d45a1561ab37d0e3cb6 (table test-table [id=58881d3f259544f09bb6295e9259719f]) successfully deleted
W20260929 22:52:50.029647 29421 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 22:52:50.029878 29421 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 22:52:50.029906 29421 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 22:52:50.031180 29421 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 22:52:50.031232 29421 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 22:52:50.031260 29421 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.147.194
I20260929 22:52:50.032922 29421 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.147.212:43149
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790722213368738-14927-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790722213368738-14927-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790722213368738-14927-0/krb5kdc/kudu_127.14.147.194.keytab
--principal=kudu/127.14.147.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.147.194:46603
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790722213368738-14927-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.14.147.194
--webserver_port=41865
--tserver_master_addrs=127.14.147.254:35819
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29421
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.147.194
--log_dir=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790722213368738-14927-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 21:58:54 UTC on ccycloud.quasar-xguoti.root.comops.site
build id 4681390
I20260929 22:52:50.033154 29421 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 22:52:50.033393 29421 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 22:52:50.038324 29433 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 22:52:50.038308 29430 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 22:52:50.038311 29431 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 22:52:50.038657 29421 server_base.cc:1034] running on GCE node
I20260929 22:52:50.038874 29421 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 22:52:50.039117 29421 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 22:52:50.040257 29421 hybrid_clock.cc:648] HybridClock initialized: now 1790722370040242 us; error 32 us; skew 500 ppm
Sep 29 22:52:50 dist-test-slave-98mk krb5kdc[28651](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790722370, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.147.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 22:52:50.042832 29421 init.cc:377] Logged in from keytab as kudu/127.14.147.194@KRBTEST.COM (short username kudu)
I20260929 22:52:50.043270 29421 webserver.cc:460] Webserver started at http://127.14.147.194:41865/ using document root <none> and password file <none>
I20260929 22:52:50.043506 29421 fs_manager.cc:359] Metadata directory not provided
I20260929 22:52:50.043567 29421 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 22:52:50.044891 29421 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260929 22:52:50.045455 29440 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 22:52:50.045608 29421 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260929 22:52:50.045684 29421 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790722213368738-14927-0/minicluster-data/ts-1/data,/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790722213368738-14927-0/minicluster-data/ts-1/wal
uuid: "ae4204ae8cb54f4f8f9a085ca5976542"
format_stamp: "Formatted at 2026-09-29 22:52:49 on dist-test-slave-98mk"
server_key: "2e9a6145dda39dc3abf6c3e9ece36713"
server_key_iv: "e70bb44c2b3731c6d82df39796783df4"
server_key_version: "encryptionkey@0"
I20260929 22:52:50.046052 29421 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790722213368738-14927-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790722213368738-14927-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790722213368738-14927-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 22:52:50.058596 29421 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 22:52:50.060607 29421 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 22:52:50.060726 29421 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 22:52:50.060971 29421 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 22:52:50.061257 29421 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 22:52:50.061290 29421 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 22:52:50.061313 29421 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 22:52:50.061337 29421 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 22:52:50.067155 29421 rpc_server.cc:307] RPC server started. Bound to: 127.14.147.194:46603
I20260929 22:52:50.067205 29553 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.147.194:46603 every 8 connection(s)
I20260929 22:52:50.067618 29421 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790722213368738-14927-0/minicluster-data/ts-1/data/info.pb
I20260929 22:52:50.070164 14927 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhWOCLk/build/release/bin/kudu as pid 29421
I20260929 22:52:50.070415 14927 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhWOCLk/build/release/bin/kudu
/tmp/dist-test-taskhWOCLk/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790722213368738-14927-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790722213368738-14927-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790722213368738-14927-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790722213368738-14927-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.147.195:41003
--local_ip_for_outbound_sockets=127.14.147.195
--tserver_master_addrs=127.14.147.254:35819
--webserver_port=32777
--webserver_interface=127.14.147.195
--keytab_file=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790722213368738-14927-0/krb5kdc/kudu_127.14.147.195.keytab
--principal=kudu/127.14.147.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.147.212:43149
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790722213368738-14927-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790722213368738-14927-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790722213368738-14927-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 29 22:52:50 dist-test-slave-98mk krb5kdc[28651](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790722370, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.147.194@KRBTEST.COM for kudu/127.14.147.254@KRBTEST.COM
I20260929 22:52:50.078574 29419 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:52:50.068312 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.147.194:53335 (local address 127.14.147.254:35819)
0929 22:52:50.068375 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0929 22:52:50.068377 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 22:52:50.068650 (+ 273us) server_negotiation.cc:407] Connection header received
0929 22:52:50.069092 (+ 442us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 22:52:50.069093 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 22:52:50.069106 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 22:52:50.069161 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0929 22:52:50.070154 (+ 993us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:52:50.070791 (+ 637us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:52:50.072196 (+ 1405us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:52:50.072504 (+ 308us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:52:50.072588 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 22:52:50.074997 (+ 2409us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 22:52:50.074999 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 22:52:50.074999 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 22:52:50.075009 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0929 22:52:50.076671 (+ 1662us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 22:52:50.077402 (+ 731us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 22:52:50.077404 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 22:52:50.077404 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 22:52:50.077479 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 22:52:50.077873 (+ 394us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 22:52:50.077874 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 22:52:50.077874 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 22:52:50.078028 (+ 154us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 22:52:50.078100 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0929 22:52:50.078361 (+ 261us) server_negotiation.cc:299] Negotiation successful
0929 22:52:50.078468 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260929 22:52:50.078663 29556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:52:50.068317 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.147.254:35819 (local address 127.14.147.194:53335)
0929 22:52:50.068479 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0929 22:52:50.068515 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0929 22:52:50.069007 (+ 492us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 22:52:50.069223 (+ 216us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 22:52:50.069226 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 22:52:50.069795 (+ 569us) client_negotiation.cc:189] Negotiated authn=SASL
0929 22:52:50.070038 (+ 243us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 22:52:50.070045 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 22:52:50.070888 (+ 843us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 22:52:50.070889 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 22:52:50.072103 (+ 1214us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 22:52:50.072106 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 22:52:50.072587 (+ 481us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 22:52:50.072587 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 22:52:50.072693 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 22:52:50.073218 (+ 525us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 22:52:50.073227 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0929 22:52:50.074878 (+ 1651us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 22:52:50.076772 (+ 1894us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 22:52:50.076780 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 22:52:50.076782 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 22:52:50.077278 (+ 496us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 22:52:50.077601 (+ 323us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 22:52:50.077602 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 22:52:50.077602 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 22:52:50.077786 (+ 184us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 22:52:50.078091 (+ 305us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 22:52:50.078094 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 22:52:50.078234 (+ 140us) client_negotiation.cc:769] Sending connection context
0929 22:52:50.078319 (+ 85us) client_negotiation.cc:240] Negotiation successful
0929 22:52:50.078476 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":85,"threads_started":1}
I20260929 22:52:50.079229 29554 heartbeater.cc:347] Connected to a master server at 127.14.147.254:35819
I20260929 22:52:50.079334 29554 heartbeater.cc:464] Registering TS with master...
I20260929 22:52:50.079556 29554 heartbeater.cc:511] Master 127.14.147.254:35819 requested a full tablet report, sending...
I20260929 22:52:50.079978 29228 ts_manager.cc:194] Registered new tserver with Master: ae4204ae8cb54f4f8f9a085ca5976542 (127.14.147.194:46603)
I20260929 22:52:50.080492 29228 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.147.194@KRBTEST.COM'} at 127.14.147.194:53335
W20260929 22:52:50.153887 29557 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 22:52:50.154105 29557 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 22:52:50.154131 29557 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 22:52:50.155320 29557 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 22:52:50.155359 29557 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 22:52:50.155387 29557 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.147.195
I20260929 22:52:50.156978 29557 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.147.212:43149
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790722213368738-14927-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790722213368738-14927-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790722213368738-14927-0/krb5kdc/kudu_127.14.147.195.keytab
--principal=kudu/127.14.147.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.147.195:41003
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790722213368738-14927-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.14.147.195
--webserver_port=32777
--tserver_master_addrs=127.14.147.254:35819
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29557
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.147.195
--log_dir=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790722213368738-14927-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 21:58:54 UTC on ccycloud.quasar-xguoti.root.comops.site
build id 4681390
I20260929 22:52:50.157198 29557 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 22:52:50.157404 29557 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 22:52:50.162336 29563 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 22:52:50.162345 29565 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 22:52:50.162336 29562 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 22:52:50.162503 29557 server_base.cc:1034] running on GCE node
I20260929 22:52:50.162855 29557 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 22:52:50.163098 29557 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 22:52:50.164276 29557 hybrid_clock.cc:648] HybridClock initialized: now 1790722370164264 us; error 55 us; skew 500 ppm
Sep 29 22:52:50 dist-test-slave-98mk krb5kdc[28651](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790722370, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.147.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 22:52:50.167058 29557 init.cc:377] Logged in from keytab as kudu/127.14.147.195@KRBTEST.COM (short username kudu)
I20260929 22:52:50.167563 29557 webserver.cc:460] Webserver started at http://127.14.147.195:32777/ using document root <none> and password file <none>
I20260929 22:52:50.167820 29557 fs_manager.cc:359] Metadata directory not provided
I20260929 22:52:50.167899 29557 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 22:52:50.169246 29557 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260929 22:52:50.169872 29572 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 22:52:50.170044 29557 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260929 22:52:50.170126 29557 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790722213368738-14927-0/minicluster-data/ts-2/data,/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790722213368738-14927-0/minicluster-data/ts-2/wal
uuid: "80e8234059154efbbc9417e8b4a71810"
format_stamp: "Formatted at 2026-09-29 22:52:49 on dist-test-slave-98mk"
server_key: "1b6941b1003aef74534e2284c46deca2"
server_key_iv: "4e9b88308cdcc5d800a9607e592d76ae"
server_key_version: "encryptionkey@0"
I20260929 22:52:50.170482 29557 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790722213368738-14927-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790722213368738-14927-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790722213368738-14927-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 22:52:50.179281 29557 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 22:52:50.181478 29557 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 22:52:50.181618 29557 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 22:52:50.181839 29557 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 22:52:50.182163 29557 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 22:52:50.182214 29557 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 22:52:50.182250 29557 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 22:52:50.182266 29557 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 22:52:50.190138 29557 rpc_server.cc:307] RPC server started. Bound to: 127.14.147.195:41003
I20260929 22:52:50.190172 29685 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.147.195:41003 every 8 connection(s)
I20260929 22:52:50.190521 29557 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790722213368738-14927-0/minicluster-data/ts-2/data/info.pb
I20260929 22:52:50.195673 14927 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhWOCLk/build/release/bin/kudu as pid 29557
Sep 29 22:52:50 dist-test-slave-98mk krb5kdc[28651](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790722370, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.147.195@KRBTEST.COM for kudu/127.14.147.254@KRBTEST.COM
I20260929 22:52:50.200675 29419 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:52:50.191220 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.147.195:59013 (local address 127.14.147.254:35819)
0929 22:52:50.191274 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0929 22:52:50.191277 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 22:52:50.191545 (+ 268us) server_negotiation.cc:407] Connection header received
0929 22:52:50.192751 (+ 1206us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 22:52:50.192753 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 22:52:50.192766 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 22:52:50.192830 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0929 22:52:50.193464 (+ 634us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:52:50.193976 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:52:50.194931 (+ 955us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:52:50.195131 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:52:50.195185 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 22:52:50.197797 (+ 2612us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 22:52:50.197798 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 22:52:50.197799 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 22:52:50.197806 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0929 22:52:50.199261 (+ 1455us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 22:52:50.199676 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 22:52:50.199677 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 22:52:50.199678 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 22:52:50.199741 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 22:52:50.199965 (+ 224us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 22:52:50.199966 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 22:52:50.199966 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 22:52:50.200098 (+ 132us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 22:52:50.200145 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0929 22:52:50.200499 (+ 354us) server_negotiation.cc:299] Negotiation successful
0929 22:52:50.200603 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260929 22:52:50.200889 29688 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:52:50.191215 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.147.254:35819 (local address 127.14.147.195:59013)
0929 22:52:50.191370 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0929 22:52:50.191436 (+ 66us) client_negotiation.cc:174] Beginning negotiation
0929 22:52:50.192638 (+ 1202us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 22:52:50.192910 (+ 272us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 22:52:50.192914 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 22:52:50.193180 (+ 266us) client_negotiation.cc:189] Negotiated authn=SASL
0929 22:52:50.193378 (+ 198us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 22:52:50.193382 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 22:52:50.194055 (+ 673us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 22:52:50.194056 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 22:52:50.194843 (+ 787us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 22:52:50.194845 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 22:52:50.195198 (+ 353us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 22:52:50.195199 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 22:52:50.195331 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 22:52:50.195897 (+ 566us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 22:52:50.195907 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0929 22:52:50.197712 (+ 1805us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 22:52:50.199332 (+ 1620us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 22:52:50.199339 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 22:52:50.199341 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 22:52:50.199609 (+ 268us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 22:52:50.199806 (+ 197us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 22:52:50.199806 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 22:52:50.199806 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 22:52:50.199904 (+ 98us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 22:52:50.200153 (+ 249us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 22:52:50.200160 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 22:52:50.200341 (+ 181us) client_negotiation.cc:769] Sending connection context
0929 22:52:50.200464 (+ 123us) client_negotiation.cc:240] Negotiation successful
0929 22:52:50.200646 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":83,"threads_started":1}
I20260929 22:52:50.201372 29686 heartbeater.cc:347] Connected to a master server at 127.14.147.254:35819
I20260929 22:52:50.201462 29686 heartbeater.cc:464] Registering TS with master...
I20260929 22:52:50.201635 29686 heartbeater.cc:511] Master 127.14.147.254:35819 requested a full tablet report, sending...
I20260929 22:52:50.202004 29228 ts_manager.cc:194] Registered new tserver with Master: 80e8234059154efbbc9417e8b4a71810 (127.14.147.195:41003)
I20260929 22:52:50.202353 29228 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.147.195@KRBTEST.COM'} at 127.14.147.195:59013
I20260929 22:52:50.202872 29690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:52:50.196054 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54112 (local address 127.14.147.254:35819)
0929 22:52:50.196143 (+ 89us) server_negotiation.cc:206] Beginning negotiation
0929 22:52:50.196145 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 22:52:50.196177 (+ 32us) server_negotiation.cc:407] Connection header received
0929 22:52:50.196251 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 22:52:50.196252 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 22:52:50.196262 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 22:52:50.196313 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0929 22:52:50.196952 (+ 639us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:52:50.197586 (+ 634us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:52:50.198541 (+ 955us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:52:50.198841 (+ 300us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:52:50.198894 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 22:52:50.200063 (+ 1169us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 22:52:50.200064 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 22:52:50.200065 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 22:52:50.200070 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0929 22:52:50.201836 (+ 1766us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 22:52:50.202143 (+ 307us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 22:52:50.202144 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 22:52:50.202144 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 22:52:50.202199 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 22:52:50.202360 (+ 161us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 22:52:50.202361 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 22:52:50.202361 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 22:52:50.202482 (+ 121us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 22:52:50.202646 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0929 22:52:50.202684 (+ 38us) server_negotiation.cc:299] Negotiation successful
0929 22:52:50.202767 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70,"thread_start_us":48,"threads_started":1}
I20260929 22:52:50.203902 14927 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 22:52:50.206734 29690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:52:50.204245 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54122 (local address 127.14.147.254:35819)
0929 22:52:50.204294 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0929 22:52:50.204296 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 22:52:50.204352 (+ 56us) server_negotiation.cc:407] Connection header received
0929 22:52:50.204431 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 22:52:50.204431 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 22:52:50.204473 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 22:52:50.204531 (+ 58us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 22:52:50.204618 (+ 87us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:52:50.205148 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:52:50.205939 (+ 791us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:52:50.206164 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:52:50.206234 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 22:52:50.206341 (+ 107us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 22:52:50.206512 (+ 171us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 22:52:50.206613 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0929 22:52:50.206631 (+ 18us) server_negotiation.cc:299] Negotiation successful
0929 22:52:50.206662 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260929 22:52:50.207077 29228 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin'} at 127.0.0.1:54122:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 22:52:50.209859 29228 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 22:52:50.212411 29350 tablet_service.cc:1467] Processing CreateTablet for tablet 984df8b1c28e43b2882f3802f7ff2238 (DEFAULT_TABLE table=test-table [id=98a4a3139de7486e827d04af6e8fd8a4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 22:52:50.212618 29350 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 984df8b1c28e43b2882f3802f7ff2238. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 22:52:50.213793 29308 tablet_bootstrap.cc:492] T 984df8b1c28e43b2882f3802f7ff2238 P 43f3886bb1b44e4aabb3e8f71ff6f3b5: Bootstrap starting.
I20260929 22:52:50.214136 29308 tablet_bootstrap.cc:654] T 984df8b1c28e43b2882f3802f7ff2238 P 43f3886bb1b44e4aabb3e8f71ff6f3b5: Neither blocks nor log segments found. Creating new log.
I20260929 22:52:50.214709 29308 tablet_bootstrap.cc:492] T 984df8b1c28e43b2882f3802f7ff2238 P 43f3886bb1b44e4aabb3e8f71ff6f3b5: No bootstrap required, opened a new log
I20260929 22:52:50.214757 29308 ts_tablet_manager.cc:1397] T 984df8b1c28e43b2882f3802f7ff2238 P 43f3886bb1b44e4aabb3e8f71ff6f3b5: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260929 22:52:50.214869 29308 raft_consensus.cc:348] T 984df8b1c28e43b2882f3802f7ff2238 P 43f3886bb1b44e4aabb3e8f71ff6f3b5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae4204ae8cb54f4f8f9a085ca5976542" member_type: VOTER last_known_addr { host: "127.14.147.194" port: 46603 } } peers { permanent_uuid: "80e8234059154efbbc9417e8b4a71810" member_type: VOTER last_known_addr { host: "127.14.147.195" port: 41003 } } peers { permanent_uuid: "43f3886bb1b44e4aabb3e8f71ff6f3b5" member_type: VOTER last_known_addr { host: "127.14.147.193" port: 46859 } }
I20260929 22:52:50.214949 29308 raft_consensus.cc:374] T 984df8b1c28e43b2882f3802f7ff2238 P 43f3886bb1b44e4aabb3e8f71ff6f3b5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 22:52:50.215000 29308 raft_consensus.cc:729] T 984df8b1c28e43b2882f3802f7ff2238 P 43f3886bb1b44e4aabb3e8f71ff6f3b5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 43f3886bb1b44e4aabb3e8f71ff6f3b5, State: Initialized, Role: FOLLOWER
I20260929 22:52:50.215042 29308 consensus_queue.cc:260] T 984df8b1c28e43b2882f3802f7ff2238 P 43f3886bb1b44e4aabb3e8f71ff6f3b5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae4204ae8cb54f4f8f9a085ca5976542" member_type: VOTER last_known_addr { host: "127.14.147.194" port: 46603 } } peers { permanent_uuid: "80e8234059154efbbc9417e8b4a71810" member_type: VOTER last_known_addr { host: "127.14.147.195" port: 41003 } } peers { permanent_uuid: "43f3886bb1b44e4aabb3e8f71ff6f3b5" member_type: VOTER last_known_addr { host: "127.14.147.193" port: 46859 } }
I20260929 22:52:50.215138 29308 ts_tablet_manager.cc:1428] T 984df8b1c28e43b2882f3802f7ff2238 P 43f3886bb1b44e4aabb3e8f71ff6f3b5: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260929 22:52:50.216120 29692 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:52:50.211788 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.147.194:46603 (local address 127.0.0.1:53956)
0929 22:52:50.212119 (+ 331us) negotiation.cc:107] Waiting for socket to connect
0929 22:52:50.212129 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0929 22:52:50.212185 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 22:52:50.212433 (+ 248us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 22:52:50.212434 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 22:52:50.212437 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 22:52:50.212482 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 22:52:50.212483 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 22:52:50.213735 (+ 1252us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 22:52:50.213736 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 22:52:50.214843 (+ 1107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 22:52:50.214847 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 22:52:50.215875 (+ 1028us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 22:52:50.215876 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 22:52:50.215939 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 22:52:50.215945 (+ 6us) client_negotiation.cc:769] Sending connection context
0929 22:52:50.215990 (+ 45us) client_negotiation.cc:240] Negotiation successful
0929 22:52:50.216036 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":315,"mutex_wait_us":43,"thread_start_us":107,"threads_started":1}
I20260929 22:52:50.216203 29423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:52:50.211776 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.147.195:41003 (local address 127.0.0.1:34178)
0929 22:52:50.211891 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0929 22:52:50.211902 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0929 22:52:50.211937 (+ 35us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 22:52:50.212364 (+ 427us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 22:52:50.212365 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 22:52:50.212367 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 22:52:50.212423 (+ 56us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 22:52:50.212425 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 22:52:50.213735 (+ 1310us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 22:52:50.213736 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 22:52:50.214883 (+ 1147us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 22:52:50.214886 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 22:52:50.215977 (+ 1091us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 22:52:50.215978 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 22:52:50.216039 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 22:52:50.216042 (+ 3us) client_negotiation.cc:769] Sending connection context
0929 22:52:50.216093 (+ 51us) client_negotiation.cc:240] Negotiation successful
0929 22:52:50.216138 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":98,"mutex_wait_us":154}
I20260929 22:52:50.216223 29694 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:52:50.212029 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34178 (local address 127.14.147.195:41003)
0929 22:52:50.212191 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0929 22:52:50.212198 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 22:52:50.212207 (+ 9us) server_negotiation.cc:407] Connection header received
0929 22:52:50.212248 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 22:52:50.212250 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 22:52:50.212266 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 22:52:50.212356 (+ 90us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 22:52:50.212482 (+ 126us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:52:50.213616 (+ 1134us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:52:50.215217 (+ 1601us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:52:50.215901 (+ 684us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:52:50.215971 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 22:52:50.216025 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0929 22:52:50.216104 (+ 79us) server_negotiation.cc:299] Negotiation successful
0929 22:52:50.216143 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":131,"thread_start_us":95,"threads_started":1}
I20260929 22:52:50.216144 29693 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:52:50.211920 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53956 (local address 127.14.147.194:46603)
0929 22:52:50.212133 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0929 22:52:50.212138 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 22:52:50.212175 (+ 37us) server_negotiation.cc:407] Connection header received
0929 22:52:50.212255 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 22:52:50.212257 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 22:52:50.212271 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 22:52:50.212354 (+ 83us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 22:52:50.212555 (+ 201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:52:50.213624 (+ 1069us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:52:50.214929 (+ 1305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:52:50.215799 (+ 870us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:52:50.215869 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 22:52:50.215913 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0929 22:52:50.216014 (+ 101us) server_negotiation.cc:299] Negotiation successful
0929 22:52:50.216057 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":173,"thread_start_us":104,"threads_started":1}
I20260929 22:52:50.216921 29613 tablet_service.cc:1467] Processing CreateTablet for tablet 984df8b1c28e43b2882f3802f7ff2238 (DEFAULT_TABLE table=test-table [id=98a4a3139de7486e827d04af6e8fd8a4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 22:52:50.217057 29488 tablet_service.cc:1467] Processing CreateTablet for tablet 984df8b1c28e43b2882f3802f7ff2238 (DEFAULT_TABLE table=test-table [id=98a4a3139de7486e827d04af6e8fd8a4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 22:52:50.217286 29613 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 984df8b1c28e43b2882f3802f7ff2238. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 22:52:50.217327 29488 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 984df8b1c28e43b2882f3802f7ff2238. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 22:52:50.221019 29696 tablet_bootstrap.cc:492] T 984df8b1c28e43b2882f3802f7ff2238 P 80e8234059154efbbc9417e8b4a71810: Bootstrap starting.
I20260929 22:52:50.221653 29696 tablet_bootstrap.cc:654] T 984df8b1c28e43b2882f3802f7ff2238 P 80e8234059154efbbc9417e8b4a71810: Neither blocks nor log segments found. Creating new log.
I20260929 22:52:50.222129 29696 log.cc:824] T 984df8b1c28e43b2882f3802f7ff2238 P 80e8234059154efbbc9417e8b4a71810: Log is configured to *not* fsync() on all Append() calls
I20260929 22:52:50.222487 29697 tablet_bootstrap.cc:492] T 984df8b1c28e43b2882f3802f7ff2238 P ae4204ae8cb54f4f8f9a085ca5976542: Bootstrap starting.
I20260929 22:52:50.223052 29697 tablet_bootstrap.cc:654] T 984df8b1c28e43b2882f3802f7ff2238 P ae4204ae8cb54f4f8f9a085ca5976542: Neither blocks nor log segments found. Creating new log.
I20260929 22:52:50.223500 29697 log.cc:824] T 984df8b1c28e43b2882f3802f7ff2238 P ae4204ae8cb54f4f8f9a085ca5976542: Log is configured to *not* fsync() on all Append() calls
I20260929 22:52:50.224143 29696 tablet_bootstrap.cc:492] T 984df8b1c28e43b2882f3802f7ff2238 P 80e8234059154efbbc9417e8b4a71810: No bootstrap required, opened a new log
I20260929 22:52:50.224216 29697 tablet_bootstrap.cc:492] T 984df8b1c28e43b2882f3802f7ff2238 P ae4204ae8cb54f4f8f9a085ca5976542: No bootstrap required, opened a new log
I20260929 22:52:50.224215 29696 ts_tablet_manager.cc:1397] T 984df8b1c28e43b2882f3802f7ff2238 P 80e8234059154efbbc9417e8b4a71810: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260929 22:52:50.224293 29697 ts_tablet_manager.cc:1397] T 984df8b1c28e43b2882f3802f7ff2238 P ae4204ae8cb54f4f8f9a085ca5976542: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260929 22:52:50.225673 29696 raft_consensus.cc:348] T 984df8b1c28e43b2882f3802f7ff2238 P 80e8234059154efbbc9417e8b4a71810 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae4204ae8cb54f4f8f9a085ca5976542" member_type: VOTER last_known_addr { host: "127.14.147.194" port: 46603 } } peers { permanent_uuid: "80e8234059154efbbc9417e8b4a71810" member_type: VOTER last_known_addr { host: "127.14.147.195" port: 41003 } } peers { permanent_uuid: "43f3886bb1b44e4aabb3e8f71ff6f3b5" member_type: VOTER last_known_addr { host: "127.14.147.193" port: 46859 } }
I20260929 22:52:50.225688 29697 raft_consensus.cc:348] T 984df8b1c28e43b2882f3802f7ff2238 P ae4204ae8cb54f4f8f9a085ca5976542 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae4204ae8cb54f4f8f9a085ca5976542" member_type: VOTER last_known_addr { host: "127.14.147.194" port: 46603 } } peers { permanent_uuid: "80e8234059154efbbc9417e8b4a71810" member_type: VOTER last_known_addr { host: "127.14.147.195" port: 41003 } } peers { permanent_uuid: "43f3886bb1b44e4aabb3e8f71ff6f3b5" member_type: VOTER last_known_addr { host: "127.14.147.193" port: 46859 } }
I20260929 22:52:50.225850 29696 raft_consensus.cc:374] T 984df8b1c28e43b2882f3802f7ff2238 P 80e8234059154efbbc9417e8b4a71810 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 22:52:50.225850 29697 raft_consensus.cc:374] T 984df8b1c28e43b2882f3802f7ff2238 P ae4204ae8cb54f4f8f9a085ca5976542 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 22:52:50.225888 29697 raft_consensus.cc:729] T 984df8b1c28e43b2882f3802f7ff2238 P ae4204ae8cb54f4f8f9a085ca5976542 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ae4204ae8cb54f4f8f9a085ca5976542, State: Initialized, Role: FOLLOWER
I20260929 22:52:50.225888 29696 raft_consensus.cc:729] T 984df8b1c28e43b2882f3802f7ff2238 P 80e8234059154efbbc9417e8b4a71810 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 80e8234059154efbbc9417e8b4a71810, State: Initialized, Role: FOLLOWER
I20260929 22:52:50.226024 29697 consensus_queue.cc:260] T 984df8b1c28e43b2882f3802f7ff2238 P ae4204ae8cb54f4f8f9a085ca5976542 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae4204ae8cb54f4f8f9a085ca5976542" member_type: VOTER last_known_addr { host: "127.14.147.194" port: 46603 } } peers { permanent_uuid: "80e8234059154efbbc9417e8b4a71810" member_type: VOTER last_known_addr { host: "127.14.147.195" port: 41003 } } peers { permanent_uuid: "43f3886bb1b44e4aabb3e8f71ff6f3b5" member_type: VOTER last_known_addr { host: "127.14.147.193" port: 46859 } }
I20260929 22:52:50.226030 29696 consensus_queue.cc:260] T 984df8b1c28e43b2882f3802f7ff2238 P 80e8234059154efbbc9417e8b4a71810 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae4204ae8cb54f4f8f9a085ca5976542" member_type: VOTER last_known_addr { host: "127.14.147.194" port: 46603 } } peers { permanent_uuid: "80e8234059154efbbc9417e8b4a71810" member_type: VOTER last_known_addr { host: "127.14.147.195" port: 41003 } } peers { permanent_uuid: "43f3886bb1b44e4aabb3e8f71ff6f3b5" member_type: VOTER last_known_addr { host: "127.14.147.193" port: 46859 } }
I20260929 22:52:50.226329 29696 ts_tablet_manager.cc:1428] T 984df8b1c28e43b2882f3802f7ff2238 P 80e8234059154efbbc9417e8b4a71810: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260929 22:52:50.226418 29686 heartbeater.cc:503] Master 127.14.147.254:35819 was elected leader, sending a full tablet report...
I20260929 22:52:50.226495 29554 heartbeater.cc:503] Master 127.14.147.254:35819 was elected leader, sending a full tablet report...
I20260929 22:52:50.226702 29697 ts_tablet_manager.cc:1428] T 984df8b1c28e43b2882f3802f7ff2238 P ae4204ae8cb54f4f8f9a085ca5976542: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
W20260929 22:52:50.318167 29555 tablet.cc:2369] T 984df8b1c28e43b2882f3802f7ff2238 P ae4204ae8cb54f4f8f9a085ca5976542: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 22:52:50.360526 29701 raft_consensus.cc:482] T 984df8b1c28e43b2882f3802f7ff2238 P 80e8234059154efbbc9417e8b4a71810 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 22:52:50.360643 29701 raft_consensus.cc:504] T 984df8b1c28e43b2882f3802f7ff2238 P 80e8234059154efbbc9417e8b4a71810 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae4204ae8cb54f4f8f9a085ca5976542" member_type: VOTER last_known_addr { host: "127.14.147.194" port: 46603 } } peers { permanent_uuid: "80e8234059154efbbc9417e8b4a71810" member_type: VOTER last_known_addr { host: "127.14.147.195" port: 41003 } } peers { permanent_uuid: "43f3886bb1b44e4aabb3e8f71ff6f3b5" member_type: VOTER last_known_addr { host: "127.14.147.193" port: 46859 } }
I20260929 22:52:50.361018 29701 leader_election.cc:288] T 984df8b1c28e43b2882f3802f7ff2238 P 80e8234059154efbbc9417e8b4a71810 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ae4204ae8cb54f4f8f9a085ca5976542 (127.14.147.194:46603), 43f3886bb1b44e4aabb3e8f71ff6f3b5 (127.14.147.193:46859)
I20260929 22:52:50.363910 29688 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:52:50.361220 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.147.193:46859 (local address 127.14.147.195:50969)
0929 22:52:50.361293 (+ 73us) negotiation.cc:107] Waiting for socket to connect
0929 22:52:50.361304 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0929 22:52:50.361345 (+ 41us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 22:52:50.361483 (+ 138us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 22:52:50.361484 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 22:52:50.361487 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 22:52:50.361558 (+ 71us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 22:52:50.361560 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 22:52:50.362293 (+ 733us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 22:52:50.362294 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 22:52:50.363064 (+ 770us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 22:52:50.363066 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 22:52:50.363676 (+ 610us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 22:52:50.363677 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 22:52:50.363759 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 22:52:50.363764 (+ 5us) client_negotiation.cc:769] Sending connection context
0929 22:52:50.363809 (+ 45us) client_negotiation.cc:240] Negotiation successful
0929 22:52:50.363850 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":53,"mutex_wait_us":1}
I20260929 22:52:50.363942 29424 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:52:50.361270 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.147.195:50969 (local address 127.14.147.193:46859)
0929 22:52:50.361321 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0929 22:52:50.361323 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 22:52:50.361334 (+ 11us) server_negotiation.cc:407] Connection header received
0929 22:52:50.361417 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 22:52:50.361418 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 22:52:50.361427 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 22:52:50.361467 (+ 40us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 22:52:50.361624 (+ 157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:52:50.362215 (+ 591us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:52:50.363150 (+ 935us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:52:50.363614 (+ 464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:52:50.363667 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 22:52:50.363675 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0929 22:52:50.363829 (+ 154us) server_negotiation.cc:299] Negotiation successful
0929 22:52:50.363869 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260929 22:52:50.364039 29702 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:52:50.361222 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.147.194:46603 (local address 127.14.147.195:51811)
0929 22:52:50.361571 (+ 349us) negotiation.cc:107] Waiting for socket to connect
0929 22:52:50.361579 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0929 22:52:50.361634 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 22:52:50.361753 (+ 119us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 22:52:50.361753 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 22:52:50.361755 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 22:52:50.361815 (+ 60us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 22:52:50.361817 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 22:52:50.362433 (+ 616us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 22:52:50.362433 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 22:52:50.363202 (+ 769us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 22:52:50.363205 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 22:52:50.363808 (+ 603us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 22:52:50.363809 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 22:52:50.363879 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 22:52:50.363883 (+ 4us) client_negotiation.cc:769] Sending connection context
0929 22:52:50.363935 (+ 52us) client_negotiation.cc:240] Negotiation successful
0929 22:52:50.363970 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":333,"mutex_wait_us":175,"thread_start_us":127,"threads_started":1}
I20260929 22:52:50.364049 29693 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:52:50.361359 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.147.195:51811 (local address 127.14.147.194:46603)
0929 22:52:50.361429 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0929 22:52:50.361431 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 22:52:50.361640 (+ 209us) server_negotiation.cc:407] Connection header received
0929 22:52:50.361702 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 22:52:50.361703 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 22:52:50.361711 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 22:52:50.361749 (+ 38us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 22:52:50.361868 (+ 119us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:52:50.362351 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:52:50.363288 (+ 937us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:52:50.363735 (+ 447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:52:50.363807 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 22:52:50.363816 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0929 22:52:50.363947 (+ 131us) server_negotiation.cc:299] Negotiation successful
0929 22:52:50.363989 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260929 22:52:50.364202 29370 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "984df8b1c28e43b2882f3802f7ff2238" candidate_uuid: "80e8234059154efbbc9417e8b4a71810" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "43f3886bb1b44e4aabb3e8f71ff6f3b5" is_pre_election: true
I20260929 22:52:50.364328 29370 raft_consensus.cc:2454] T 984df8b1c28e43b2882f3802f7ff2238 P 43f3886bb1b44e4aabb3e8f71ff6f3b5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 80e8234059154efbbc9417e8b4a71810 in term 0.
I20260929 22:52:50.364517 29573 leader_election.cc:302] T 984df8b1c28e43b2882f3802f7ff2238 P 80e8234059154efbbc9417e8b4a71810 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 43f3886bb1b44e4aabb3e8f71ff6f3b5, 80e8234059154efbbc9417e8b4a71810; no voters:
I20260929 22:52:50.364653 29701 raft_consensus.cc:2781] T 984df8b1c28e43b2882f3802f7ff2238 P 80e8234059154efbbc9417e8b4a71810 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 22:52:50.364668 29508 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "984df8b1c28e43b2882f3802f7ff2238" candidate_uuid: "80e8234059154efbbc9417e8b4a71810" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ae4204ae8cb54f4f8f9a085ca5976542" is_pre_election: true
I20260929 22:52:50.364727 29701 raft_consensus.cc:482] T 984df8b1c28e43b2882f3802f7ff2238 P 80e8234059154efbbc9417e8b4a71810 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 22:52:50.364753 29701 raft_consensus.cc:3037] T 984df8b1c28e43b2882f3802f7ff2238 P 80e8234059154efbbc9417e8b4a71810 [term 0 FOLLOWER]: Advancing to term 1
I20260929 22:52:50.364782 29508 raft_consensus.cc:2454] T 984df8b1c28e43b2882f3802f7ff2238 P ae4204ae8cb54f4f8f9a085ca5976542 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 80e8234059154efbbc9417e8b4a71810 in term 0.
I20260929 22:52:50.365486 29701 raft_consensus.cc:504] T 984df8b1c28e43b2882f3802f7ff2238 P 80e8234059154efbbc9417e8b4a71810 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae4204ae8cb54f4f8f9a085ca5976542" member_type: VOTER last_known_addr { host: "127.14.147.194" port: 46603 } } peers { permanent_uuid: "80e8234059154efbbc9417e8b4a71810" member_type: VOTER last_known_addr { host: "127.14.147.195" port: 41003 } } peers { permanent_uuid: "43f3886bb1b44e4aabb3e8f71ff6f3b5" member_type: VOTER last_known_addr { host: "127.14.147.193" port: 46859 } }
I20260929 22:52:50.365612 29701 leader_election.cc:288] T 984df8b1c28e43b2882f3802f7ff2238 P 80e8234059154efbbc9417e8b4a71810 [CANDIDATE]: Term 1 election: Requested vote from peers ae4204ae8cb54f4f8f9a085ca5976542 (127.14.147.194:46603), 43f3886bb1b44e4aabb3e8f71ff6f3b5 (127.14.147.193:46859)
I20260929 22:52:50.365762 29508 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "984df8b1c28e43b2882f3802f7ff2238" candidate_uuid: "80e8234059154efbbc9417e8b4a71810" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ae4204ae8cb54f4f8f9a085ca5976542"
I20260929 22:52:50.365840 29508 raft_consensus.cc:3037] T 984df8b1c28e43b2882f3802f7ff2238 P ae4204ae8cb54f4f8f9a085ca5976542 [term 0 FOLLOWER]: Advancing to term 1
I20260929 22:52:50.366549 29508 raft_consensus.cc:2454] T 984df8b1c28e43b2882f3802f7ff2238 P ae4204ae8cb54f4f8f9a085ca5976542 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 80e8234059154efbbc9417e8b4a71810 in term 1.
I20260929 22:52:50.366652 29370 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "984df8b1c28e43b2882f3802f7ff2238" candidate_uuid: "80e8234059154efbbc9417e8b4a71810" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "43f3886bb1b44e4aabb3e8f71ff6f3b5"
I20260929 22:52:50.366708 29574 leader_election.cc:302] T 984df8b1c28e43b2882f3802f7ff2238 P 80e8234059154efbbc9417e8b4a71810 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 80e8234059154efbbc9417e8b4a71810, ae4204ae8cb54f4f8f9a085ca5976542; no voters:
I20260929 22:52:50.366730 29370 raft_consensus.cc:3037] T 984df8b1c28e43b2882f3802f7ff2238 P 43f3886bb1b44e4aabb3e8f71ff6f3b5 [term 0 FOLLOWER]: Advancing to term 1
I20260929 22:52:50.366792 29701 raft_consensus.cc:2781] T 984df8b1c28e43b2882f3802f7ff2238 P 80e8234059154efbbc9417e8b4a71810 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 22:52:50.366968 29701 raft_consensus.cc:686] T 984df8b1c28e43b2882f3802f7ff2238 P 80e8234059154efbbc9417e8b4a71810 [term 1 LEADER]: Becoming Leader. State: Replica: 80e8234059154efbbc9417e8b4a71810, State: Running, Role: LEADER
I20260929 22:52:50.367079 29701 consensus_queue.cc:237] T 984df8b1c28e43b2882f3802f7ff2238 P 80e8234059154efbbc9417e8b4a71810 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae4204ae8cb54f4f8f9a085ca5976542" member_type: VOTER last_known_addr { host: "127.14.147.194" port: 46603 } } peers { permanent_uuid: "80e8234059154efbbc9417e8b4a71810" member_type: VOTER last_known_addr { host: "127.14.147.195" port: 41003 } } peers { permanent_uuid: "43f3886bb1b44e4aabb3e8f71ff6f3b5" member_type: VOTER last_known_addr { host: "127.14.147.193" port: 46859 } }
I20260929 22:52:50.368000 29370 raft_consensus.cc:2454] T 984df8b1c28e43b2882f3802f7ff2238 P 43f3886bb1b44e4aabb3e8f71ff6f3b5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 80e8234059154efbbc9417e8b4a71810 in term 1.
I20260929 22:52:50.368028 29228 catalog_manager.cc:5495] T 984df8b1c28e43b2882f3802f7ff2238 P 80e8234059154efbbc9417e8b4a71810 reported cstate change: term changed from 0 to 1, leader changed from <none> to 80e8234059154efbbc9417e8b4a71810 (127.14.147.195). New cstate: current_term: 1 leader_uuid: "80e8234059154efbbc9417e8b4a71810" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae4204ae8cb54f4f8f9a085ca5976542" member_type: VOTER last_known_addr { host: "127.14.147.194" port: 46603 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "80e8234059154efbbc9417e8b4a71810" member_type: VOTER last_known_addr { host: "127.14.147.195" port: 41003 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "43f3886bb1b44e4aabb3e8f71ff6f3b5" member_type: VOTER last_known_addr { host: "127.14.147.193" port: 46859 } health_report { overall_health: UNKNOWN } } }
I20260929 22:52:50.400568 29694 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:52:50.398268 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34184 (local address 127.14.147.195:41003)
0929 22:52:50.398347 (+ 79us) server_negotiation.cc:206] Beginning negotiation
0929 22:52:50.398349 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 22:52:50.398357 (+ 8us) server_negotiation.cc:407] Connection header received
0929 22:52:50.398396 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 22:52:50.398396 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 22:52:50.398415 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 22:52:50.398482 (+ 67us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 22:52:50.398583 (+ 101us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:52:50.399150 (+ 567us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:52:50.399869 (+ 719us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:52:50.400054 (+ 185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:52:50.400101 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 22:52:50.400286 (+ 185us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 22:52:50.400393 (+ 107us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 22:52:50.400453 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0929 22:52:50.400468 (+ 15us) server_negotiation.cc:299] Negotiation successful
0929 22:52:50.400488 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260929 22:52:50.402348 29370 raft_consensus.cc:1261] T 984df8b1c28e43b2882f3802f7ff2238 P 43f3886bb1b44e4aabb3e8f71ff6f3b5 [term 1 FOLLOWER]: Refusing update from remote peer 80e8234059154efbbc9417e8b4a71810: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 22:52:50.402349 29508 raft_consensus.cc:1261] T 984df8b1c28e43b2882f3802f7ff2238 P ae4204ae8cb54f4f8f9a085ca5976542 [term 1 FOLLOWER]: Refusing update from remote peer 80e8234059154efbbc9417e8b4a71810: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 22:52:50.402711 29703 consensus_queue.cc:1035] T 984df8b1c28e43b2882f3802f7ff2238 P 80e8234059154efbbc9417e8b4a71810 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae4204ae8cb54f4f8f9a085ca5976542" member_type: VOTER last_known_addr { host: "127.14.147.194" port: 46603 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 22:52:50.402853 29701 consensus_queue.cc:1035] T 984df8b1c28e43b2882f3802f7ff2238 P 80e8234059154efbbc9417e8b4a71810 [LEADER]: Connected to new peer: Peer: permanent_uuid: "43f3886bb1b44e4aabb3e8f71ff6f3b5" member_type: VOTER last_known_addr { host: "127.14.147.193" port: 46859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 22:52:50.405668 29708 mvcc.cc:205] Tried to move back new op lower bound from 7334798829165441024 to 7334798829024296960. Current Snapshot: MvccSnapshot[applied={T|T < 7334798829165441024}]
I20260929 22:52:50.406999 29228 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin'} at 127.0.0.1:54122:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 22:52:50.407079 29228 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin'} at 127.0.0.1:54122:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 22:52:50.407739 29228 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 83dd29106e1946398652603810176095: Sending DeleteTablet for 3 replicas of tablet 984df8b1c28e43b2882f3802f7ff2238
I20260929 22:52:50.408017 29350 tablet_service.cc:1514] Processing DeleteTablet for tablet 984df8b1c28e43b2882f3802f7ff2238 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 22:52:50 UTC) from {username='kudu', principal='kudu/127.14.147.254@KRBTEST.COM'} at 127.0.0.1:58754
I20260929 22:52:50.408007 29613 tablet_service.cc:1514] Processing DeleteTablet for tablet 984df8b1c28e43b2882f3802f7ff2238 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 22:52:50 UTC) from {username='kudu', principal='kudu/127.14.147.254@KRBTEST.COM'} at 127.0.0.1:34178
I20260929 22:52:50.408013 29488 tablet_service.cc:1514] Processing DeleteTablet for tablet 984df8b1c28e43b2882f3802f7ff2238 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 22:52:50 UTC) from {username='kudu', principal='kudu/127.14.147.254@KRBTEST.COM'} at 127.0.0.1:53956
I20260929 22:52:50.408098 29425 tablet_replica.cc:330] T 984df8b1c28e43b2882f3802f7ff2238 P 43f3886bb1b44e4aabb3e8f71ff6f3b5: stopping tablet replica
I20260929 22:52:50.408142 29425 raft_consensus.cc:2229] T 984df8b1c28e43b2882f3802f7ff2238 P 43f3886bb1b44e4aabb3e8f71ff6f3b5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 22:52:50.408180 29425 raft_consensus.cc:2258] T 984df8b1c28e43b2882f3802f7ff2238 P 43f3886bb1b44e4aabb3e8f71ff6f3b5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 22:52:50.408401 29715 tablet_replica.cc:330] T 984df8b1c28e43b2882f3802f7ff2238 P ae4204ae8cb54f4f8f9a085ca5976542: stopping tablet replica
I20260929 22:52:50.408398 14927 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhWOCLk/build/release/bin/kudu with pid 29266
I20260929 22:52:50.408493 29715 raft_consensus.cc:2229] T 984df8b1c28e43b2882f3802f7ff2238 P ae4204ae8cb54f4f8f9a085ca5976542 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 22:52:50.408497 29425 ts_tablet_manager.cc:1905] T 984df8b1c28e43b2882f3802f7ff2238 P 43f3886bb1b44e4aabb3e8f71ff6f3b5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 22:52:50.408579 29715 raft_consensus.cc:2258] T 984df8b1c28e43b2882f3802f7ff2238 P ae4204ae8cb54f4f8f9a085ca5976542 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 22:52:50.408905 29716 tablet_replica.cc:330] T 984df8b1c28e43b2882f3802f7ff2238 P 80e8234059154efbbc9417e8b4a71810: stopping tablet replica
I20260929 22:52:50.408985 29716 raft_consensus.cc:2229] T 984df8b1c28e43b2882f3802f7ff2238 P 80e8234059154efbbc9417e8b4a71810 [term 1 LEADER]: Raft consensus shutting down.
I20260929 22:52:50.408987 29715 ts_tablet_manager.cc:1905] T 984df8b1c28e43b2882f3802f7ff2238 P ae4204ae8cb54f4f8f9a085ca5976542: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 22:52:50.409065 29716 raft_consensus.cc:2258] T 984df8b1c28e43b2882f3802f7ff2238 P 80e8234059154efbbc9417e8b4a71810 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 22:52:50.410806 29715 ts_tablet_manager.cc:1918] T 984df8b1c28e43b2882f3802f7ff2238 P ae4204ae8cb54f4f8f9a085ca5976542: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 22:52:50.410863 29715 log.cc:1196] T 984df8b1c28e43b2882f3802f7ff2238 P ae4204ae8cb54f4f8f9a085ca5976542: Deleting WAL directory at /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790722213368738-14927-0/minicluster-data/ts-1/wal/wals/984df8b1c28e43b2882f3802f7ff2238
I20260929 22:52:50.411114 29715 ts_tablet_manager.cc:1939] T 984df8b1c28e43b2882f3802f7ff2238 P ae4204ae8cb54f4f8f9a085ca5976542: Deleting consensus metadata
I20260929 22:52:50.411480 29213 catalog_manager.cc:4841] TS ae4204ae8cb54f4f8f9a085ca5976542 (127.14.147.194:46603): tablet 984df8b1c28e43b2882f3802f7ff2238 (table test-table [id=98a4a3139de7486e827d04af6e8fd8a4]) successfully deleted
I20260929 22:52:50.415730 29716 ts_tablet_manager.cc:1905] T 984df8b1c28e43b2882f3802f7ff2238 P 80e8234059154efbbc9417e8b4a71810: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260929 22:52:50.416348 29212 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.14.147.193:46859 (error 108)
I20260929 22:52:50.416826 14927 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhWOCLk/build/release/bin/kudu with pid 29421
I20260929 22:52:50.416939 29692 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:52:50.416816 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.147.193:46859 (local address 127.0.0.1:58760)
0929 22:52:50.416869 (+ 53us) negotiation.cc:107] Waiting for socket to connect
0929 22:52:50.416905 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.147.193:46859: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
W20260929 22:52:50.417507 29212 catalog_manager.cc:4584] TS 43f3886bb1b44e4aabb3e8f71ff6f3b5 (127.14.147.193:46859): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 984df8b1c28e43b2882f3802f7ff2238: Network error: Client connection negotiation failed: client connection to 127.14.147.193:46859: connect: Connection refused (error 111)
I20260929 22:52:50.417989 29716 ts_tablet_manager.cc:1918] T 984df8b1c28e43b2882f3802f7ff2238 P 80e8234059154efbbc9417e8b4a71810: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 22:52:50.418056 29716 log.cc:1196] T 984df8b1c28e43b2882f3802f7ff2238 P 80e8234059154efbbc9417e8b4a71810: Deleting WAL directory at /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790722213368738-14927-0/minicluster-data/ts-2/wal/wals/984df8b1c28e43b2882f3802f7ff2238
I20260929 22:52:50.418277 29716 ts_tablet_manager.cc:1939] T 984df8b1c28e43b2882f3802f7ff2238 P 80e8234059154efbbc9417e8b4a71810: Deleting consensus metadata
I20260929 22:52:50.418635 29212 catalog_manager.cc:4841] TS 80e8234059154efbbc9417e8b4a71810 (127.14.147.195:41003): tablet 984df8b1c28e43b2882f3802f7ff2238 (table test-table [id=98a4a3139de7486e827d04af6e8fd8a4]) successfully deleted
I20260929 22:52:50.422416 14927 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhWOCLk/build/release/bin/kudu with pid 29557
I20260929 22:52:50.426349 14927 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhWOCLk/build/release/bin/kudu with pid 29195
I20260929 22:52:50.430783 14927 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhWOCLk/build/release/bin/kudu
/tmp/dist-test-taskhWOCLk/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790722213368738-14927-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790722213368738-14927-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790722213368738-14927-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790722213368738-14927-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.147.254:35819
--webserver_interface=127.14.147.254
--webserver_port=33785
--keytab_file=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790722213368738-14927-0/krb5kdc/kudu_127.14.147.254.keytab
--principal=kudu/127.14.147.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.147.212:43149
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.147.254:35819
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790722213368738-14927-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790722213368738-14927-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790722213368738-14927-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 22:52:50.511468 29718 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 22:52:50.511693 29718 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 22:52:50.511731 29718 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 22:52:50.512933 29718 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 22:52:50.512974 29718 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 22:52:50.512992 29718 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 22:52:50.513007 29718 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 22:52:50.513023 29718 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 22:52:50.514557 29718 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.147.212:43149
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790722213368738-14927-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790722213368738-14927-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.14.147.254:35819
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790722213368738-14927-0/krb5kdc/kudu_127.14.147.254.keytab
--principal=kudu/127.14.147.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.147.254:35819
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790722213368738-14927-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.14.147.254
--webserver_port=33785
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29718
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790722213368738-14927-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 21:58:54 UTC on ccycloud.quasar-xguoti.root.comops.site
build id 4681390
I20260929 22:52:50.514816 29718 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 22:52:50.515030 29718 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260929 22:52:50.519687 29718 server_base.cc:1034] running on GCE node
W20260929 22:52:50.519646 29726 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 22:52:50.519632 29723 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 22:52:50.519655 29724 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 22:52:50.520030 29718 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 22:52:50.520304 29718 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 22:52:50.521445 29718 hybrid_clock.cc:648] HybridClock initialized: now 1790722370521436 us; error 27 us; skew 500 ppm
Sep 29 22:52:50 dist-test-slave-98mk krb5kdc[28651](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790722370, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.147.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 22:52:50.523867 29718 init.cc:377] Logged in from keytab as kudu/127.14.147.254@KRBTEST.COM (short username kudu)
I20260929 22:52:50.524294 29718 webserver.cc:460] Webserver started at http://127.14.147.254:33785/ using document root <none> and password file <none>
I20260929 22:52:50.524513 29718 fs_manager.cc:359] Metadata directory not provided
I20260929 22:52:50.524559 29718 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 22:52:50.525858 29718 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260929 22:52:50.526434 29733 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 22:52:50.526589 29718 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260929 22:52:50.526669 29718 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790722213368738-14927-0/minicluster-data/master-0/data,/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790722213368738-14927-0/minicluster-data/master-0/wal
uuid: "83dd29106e1946398652603810176095"
format_stamp: "Formatted at 2026-09-29 22:52:48 on dist-test-slave-98mk"
server_key: "dfa6a553431dbd9d21657950be5ff375"
server_key_iv: "dee84b8c1d392b6d5fef2d8ffe935993"
server_key_version: "encryptionkey@0"
I20260929 22:52:50.527010 29718 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790722213368738-14927-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790722213368738-14927-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790722213368738-14927-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 22:52:50.551528 29718 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 22:52:50.553496 29718 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 22:52:50.553644 29718 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 22:52:50.557670 29718 rpc_server.cc:307] RPC server started. Bound to: 127.14.147.254:35819
I20260929 22:52:50.557711 29785 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.147.254:35819 every 8 connection(s)
I20260929 22:52:50.558069 29718 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790722213368738-14927-0/minicluster-data/master-0/data/info.pb
I20260929 22:52:50.559582 29786 sys_catalog.cc:263] Verifying existing consensus state
I20260929 22:52:50.560222 29786 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 83dd29106e1946398652603810176095: Bootstrap starting.
I20260929 22:52:50.565907 14927 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhWOCLk/build/release/bin/kudu as pid 29718
I20260929 22:52:50.566185 14927 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhWOCLk/build/release/bin/kudu
/tmp/dist-test-taskhWOCLk/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790722213368738-14927-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790722213368738-14927-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790722213368738-14927-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790722213368738-14927-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.147.193:46859
--local_ip_for_outbound_sockets=127.14.147.193
--tserver_master_addrs=127.14.147.254:35819
--webserver_port=35563
--webserver_interface=127.14.147.193
--keytab_file=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790722213368738-14927-0/krb5kdc/kudu_127.14.147.193.keytab
--principal=kudu/127.14.147.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.147.212:43149
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790722213368738-14927-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790722213368738-14927-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790722213368738-14927-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 22:52:50.568537 29786 log.cc:824] T 00000000000000000000000000000000 P 83dd29106e1946398652603810176095: Log is configured to *not* fsync() on all Append() calls
I20260929 22:52:50.570777 29786 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 83dd29106e1946398652603810176095: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=7 ignored=0} mutations{seen=9 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260929 22:52:50.571061 29786 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 83dd29106e1946398652603810176095: Bootstrap complete.
I20260929 22:52:50.573410 29786 raft_consensus.cc:348] T 00000000000000000000000000000000 P 83dd29106e1946398652603810176095 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "83dd29106e1946398652603810176095" member_type: VOTER last_known_addr { host: "127.14.147.254" port: 35819 } }
I20260929 22:52:50.573644 29786 raft_consensus.cc:729] T 00000000000000000000000000000000 P 83dd29106e1946398652603810176095 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 83dd29106e1946398652603810176095, State: Initialized, Role: FOLLOWER
I20260929 22:52:50.573797 29786 consensus_queue.cc:260] T 00000000000000000000000000000000 P 83dd29106e1946398652603810176095 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "83dd29106e1946398652603810176095" member_type: VOTER last_known_addr { host: "127.14.147.254" port: 35819 } }
I20260929 22:52:50.573871 29786 raft_consensus.cc:388] T 00000000000000000000000000000000 P 83dd29106e1946398652603810176095 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 22:52:50.573920 29786 raft_consensus.cc:482] T 00000000000000000000000000000000 P 83dd29106e1946398652603810176095 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 22:52:50.573966 29786 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 83dd29106e1946398652603810176095 [term 2 FOLLOWER]: Advancing to term 3
I20260929 22:52:50.574970 29786 raft_consensus.cc:504] T 00000000000000000000000000000000 P 83dd29106e1946398652603810176095 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "83dd29106e1946398652603810176095" member_type: VOTER last_known_addr { host: "127.14.147.254" port: 35819 } }
I20260929 22:52:50.575070 29786 leader_election.cc:302] T 00000000000000000000000000000000 P 83dd29106e1946398652603810176095 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 83dd29106e1946398652603810176095; no voters:
I20260929 22:52:50.575239 29786 leader_election.cc:288] T 00000000000000000000000000000000 P 83dd29106e1946398652603810176095 [CANDIDATE]: Term 3 election: Requested vote from peers
I20260929 22:52:50.575289 29790 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 83dd29106e1946398652603810176095 [term 3 FOLLOWER]: Leader election won for term 3
I20260929 22:52:50.575398 29790 raft_consensus.cc:686] T 00000000000000000000000000000000 P 83dd29106e1946398652603810176095 [term 3 LEADER]: Becoming Leader. State: Replica: 83dd29106e1946398652603810176095, State: Running, Role: LEADER
I20260929 22:52:50.575539 29786 sys_catalog.cc:564] T 00000000000000000000000000000000 P 83dd29106e1946398652603810176095 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 22:52:50.575532 29790 consensus_queue.cc:237] T 00000000000000000000000000000000 P 83dd29106e1946398652603810176095 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "83dd29106e1946398652603810176095" member_type: VOTER last_known_addr { host: "127.14.147.254" port: 35819 } }
I20260929 22:52:50.576443 29791 sys_catalog.cc:455] T 00000000000000000000000000000000 P 83dd29106e1946398652603810176095 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "83dd29106e1946398652603810176095" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "83dd29106e1946398652603810176095" member_type: VOTER last_known_addr { host: "127.14.147.254" port: 35819 } } }
I20260929 22:52:50.576460 29792 sys_catalog.cc:455] T 00000000000000000000000000000000 P 83dd29106e1946398652603810176095 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 83dd29106e1946398652603810176095. Latest consensus state: current_term: 3 leader_uuid: "83dd29106e1946398652603810176095" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "83dd29106e1946398652603810176095" member_type: VOTER last_known_addr { host: "127.14.147.254" port: 35819 } } }
I20260929 22:52:50.576545 29791 sys_catalog.cc:458] T 00000000000000000000000000000000 P 83dd29106e1946398652603810176095 [sys.catalog]: This master's current role is: LEADER
I20260929 22:52:50.576558 29792 sys_catalog.cc:458] T 00000000000000000000000000000000 P 83dd29106e1946398652603810176095 [sys.catalog]: This master's current role is: LEADER
I20260929 22:52:50.576757 29802 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 22:52:50.577562 29802 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 22:52:50.577663 29802 catalog_manager.cc:1259] Loaded cluster ID: 56089a3d64ff483fa50c5dd79b352240
I20260929 22:52:50.577692 29802 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
F20260929 22:52:50.577901 29802 catalog_manager.cc:1466] Initializing Kudu internal certificate authority failed: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7f01f29379ad google::LogMessage::Fail() at ??:0
@ 0x7f01f293a88a google::LogMessage::SendToLog() at ??:0
@ 0x7f01f29374de google::LogMessage::Flush() at ??:0
@ 0x7f01f2938a19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f01fa8580f1 _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlSt8functionIFNS_6StatusEvEERKNS_9consensus13RaftConsensusElPKcE_clES5_S9_lSB_.isra.591 at ??:0
@ 0x7f01fa8618b9 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7f01f48050d3 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f01f47fa954 kudu::Thread::SuperviseThread() at ??:0
@ 0x7f01f565e6db start_thread at ??:0
@ 0x7f01f143561f clone at ??:0
W20260929 22:52:50.660046 29788 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 22:52:50.660264 29788 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 22:52:50.660290 29788 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 22:52:50.661613 29788 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 22:52:50.661661 29788 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 22:52:50.661689 29788 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.147.193
I20260929 22:52:50.663244 29788 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.147.212:43149
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790722213368738-14927-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790722213368738-14927-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790722213368738-14927-0/krb5kdc/kudu_127.14.147.193.keytab
--principal=kudu/127.14.147.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.147.193:46859
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790722213368738-14927-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.14.147.193
--webserver_port=35563
--tserver_master_addrs=127.14.147.254:35819
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29788
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.147.193
--log_dir=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790722213368738-14927-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 21:58:54 UTC on ccycloud.quasar-xguoti.root.comops.site
build id 4681390
I20260929 22:52:50.663518 29788 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 22:52:50.663734 29788 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 22:52:50.668504 29822 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 22:52:50.668526 29825 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 22:52:50.668532 29823 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 22:52:50.668756 29788 server_base.cc:1034] running on GCE node
I20260929 22:52:50.668998 29788 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 22:52:50.669284 29788 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 22:52:50.670468 29788 hybrid_clock.cc:648] HybridClock initialized: now 1790722370670450 us; error 32 us; skew 500 ppm
Sep 29 22:52:50 dist-test-slave-98mk krb5kdc[28651](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790722370, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.147.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 22:52:50.673523 29788 init.cc:377] Logged in from keytab as kudu/127.14.147.193@KRBTEST.COM (short username kudu)
I20260929 22:52:50.674100 29788 webserver.cc:460] Webserver started at http://127.14.147.193:35563/ using document root <none> and password file <none>
I20260929 22:52:50.674387 29788 fs_manager.cc:359] Metadata directory not provided
I20260929 22:52:50.674458 29788 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 22:52:50.676218 29788 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.002s sys 0.000s
I20260929 22:52:50.677057 29832 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 22:52:50.677237 29788 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260929 22:52:50.677335 29788 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790722213368738-14927-0/minicluster-data/ts-0/data,/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790722213368738-14927-0/minicluster-data/ts-0/wal
uuid: "43f3886bb1b44e4aabb3e8f71ff6f3b5"
format_stamp: "Formatted at 2026-09-29 22:52:49 on dist-test-slave-98mk"
server_key: "85d4f73702f5e7e081d2846fbdda5227"
server_key_iv: "933fc3cece9f429af97ebdb9a876d23e"
server_key_version: "encryptionkey@0"
I20260929 22:52:50.677798 29788 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790722213368738-14927-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790722213368738-14927-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790722213368738-14927-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 22:52:50.706948 29788 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 22:52:50.709650 29788 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 22:52:50.709822 29788 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 22:52:50.710095 29788 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 22:52:50.710628 29839 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260929 22:52:50.712158 29788 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260929 22:52:50.712230 29788 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260929 22:52:50.712265 29788 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260929 22:52:50.713145 29788 ts_tablet_manager.cc:610] Registered 1 tablets
I20260929 22:52:50.713191 29788 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20260929 22:52:50.713425 29839 tablet_bootstrap.cc:492] T 984df8b1c28e43b2882f3802f7ff2238 P 43f3886bb1b44e4aabb3e8f71ff6f3b5: Bootstrap starting.
I20260929 22:52:50.715543 29839 log.cc:824] T 984df8b1c28e43b2882f3802f7ff2238 P 43f3886bb1b44e4aabb3e8f71ff6f3b5: Log is configured to *not* fsync() on all Append() calls
I20260929 22:52:50.718501 29839 tablet_bootstrap.cc:492] T 984df8b1c28e43b2882f3802f7ff2238 P 43f3886bb1b44e4aabb3e8f71ff6f3b5: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260929 22:52:50.718796 29839 tablet_bootstrap.cc:492] T 984df8b1c28e43b2882f3802f7ff2238 P 43f3886bb1b44e4aabb3e8f71ff6f3b5: Bootstrap complete.
I20260929 22:52:50.718966 29839 ts_tablet_manager.cc:1397] T 984df8b1c28e43b2882f3802f7ff2238 P 43f3886bb1b44e4aabb3e8f71ff6f3b5: Time spent bootstrapping tablet: real 0.006s user 0.003s sys 0.000s
I20260929 22:52:50.720242 29788 rpc_server.cc:307] RPC server started. Bound to: 127.14.147.193:46859
I20260929 22:52:50.720355 29947 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.147.193:46859 every 8 connection(s)
I20260929 22:52:50.720616 29788 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790722213368738-14927-0/minicluster-data/ts-0/data/info.pb
I20260929 22:52:50.721444 14927 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhWOCLk/build/release/bin/kudu as pid 29788
I20260929 22:52:50.720741 29839 raft_consensus.cc:348] T 984df8b1c28e43b2882f3802f7ff2238 P 43f3886bb1b44e4aabb3e8f71ff6f3b5 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae4204ae8cb54f4f8f9a085ca5976542" member_type: VOTER last_known_addr { host: "127.14.147.194" port: 46603 } } peers { permanent_uuid: "80e8234059154efbbc9417e8b4a71810" member_type: VOTER last_known_addr { host: "127.14.147.195" port: 41003 } } peers { permanent_uuid: "43f3886bb1b44e4aabb3e8f71ff6f3b5" member_type: VOTER last_known_addr { host: "127.14.147.193" port: 46859 } }
I20260929 22:52:50.721541 29839 raft_consensus.cc:729] T 984df8b1c28e43b2882f3802f7ff2238 P 43f3886bb1b44e4aabb3e8f71ff6f3b5 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 43f3886bb1b44e4aabb3e8f71ff6f3b5, State: Initialized, Role: FOLLOWER
I20260929 22:52:50.721700 14927 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhWOCLk/build/release/bin/kudu
/tmp/dist-test-taskhWOCLk/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790722213368738-14927-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790722213368738-14927-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790722213368738-14927-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790722213368738-14927-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.147.194:46603
--local_ip_for_outbound_sockets=127.14.147.194
--tserver_master_addrs=127.14.147.254:35819
--webserver_port=41865
--webserver_interface=127.14.147.194
--keytab_file=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790722213368738-14927-0/krb5kdc/kudu_127.14.147.194.keytab
--principal=kudu/127.14.147.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.147.212:43149
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790722213368738-14927-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790722213368738-14927-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790722213368738-14927-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 22:52:50.721684 29839 consensus_queue.cc:260] T 984df8b1c28e43b2882f3802f7ff2238 P 43f3886bb1b44e4aabb3e8f71ff6f3b5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae4204ae8cb54f4f8f9a085ca5976542" member_type: VOTER last_known_addr { host: "127.14.147.194" port: 46603 } } peers { permanent_uuid: "80e8234059154efbbc9417e8b4a71810" member_type: VOTER last_known_addr { host: "127.14.147.195" port: 41003 } } peers { permanent_uuid: "43f3886bb1b44e4aabb3e8f71ff6f3b5" member_type: VOTER last_known_addr { host: "127.14.147.193" port: 46859 } }
I20260929 22:52:50.721946 29839 ts_tablet_manager.cc:1428] T 984df8b1c28e43b2882f3802f7ff2238 P 43f3886bb1b44e4aabb3e8f71ff6f3b5: Time spent starting tablet: real 0.003s user 0.002s sys 0.003s
W20260929 22:52:50.767921 29950 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 22:52:50.721288 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.147.254:35819 (local address 127.14.147.193:52323)
0929 22:52:50.721481 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0929 22:52:50.721533 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0929 22:52:50.721718 (+ 185us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 22:52:50.767702 (+ 45984us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.147.254:35819: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":145,"thread_start_us":90,"threads_started":1}
W20260929 22:52:50.768195 29948 heartbeater.cc:644] Failed to heartbeat to 127.14.147.254:35819 (0 consecutive failures): Network error: Failed to ping master at 127.14.147.254:35819: Client connection negotiation failed: client connection to 127.14.147.254:35819: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260929 22:52:50.768534 29950 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:52:50.768418 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.147.254:35819 (local address 127.14.147.193:58741)
0929 22:52:50.768468 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0929 22:52:50.768512 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.147.254:35819: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
I20260929 22:52:50.768878 29950 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:52:50.768806 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.147.254:35819 (local address 127.14.147.193:40223)
0929 22:52:50.768839 (+ 33us) negotiation.cc:107] Waiting for socket to connect
0929 22:52:50.768861 (+ 22us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.147.254:35819: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":23}
W20260929 22:52:50.768996 29948 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260929 22:52:50.804144 29952 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 22:52:50.804375 29952 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 22:52:50.804402 29952 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 22:52:50.805774 29952 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 22:52:50.805841 29952 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 22:52:50.805872 29952 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.147.194
I20260929 22:52:50.807502 29952 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.147.212:43149
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790722213368738-14927-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790722213368738-14927-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790722213368738-14927-0/krb5kdc/kudu_127.14.147.194.keytab
--principal=kudu/127.14.147.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.147.194:46603
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790722213368738-14927-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.14.147.194
--webserver_port=41865
--tserver_master_addrs=127.14.147.254:35819
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29952
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.147.194
--log_dir=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790722213368738-14927-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 21:58:54 UTC on ccycloud.quasar-xguoti.root.comops.site
build id 4681390
I20260929 22:52:50.807717 29952 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 22:52:50.807914 29952 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 22:52:50.812785 29957 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 22:52:50.812860 29952 server_base.cc:1034] running on GCE node
W20260929 22:52:50.812810 29960 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 22:52:50.812785 29958 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 22:52:50.813162 29952 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 22:52:50.813412 29952 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 22:52:50.814586 29952 hybrid_clock.cc:648] HybridClock initialized: now 1790722370814566 us; error 30 us; skew 500 ppm
Sep 29 22:52:50 dist-test-slave-98mk krb5kdc[28651](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790722370, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.147.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 22:52:50.817198 29952 init.cc:377] Logged in from keytab as kudu/127.14.147.194@KRBTEST.COM (short username kudu)
I20260929 22:52:50.817638 29952 webserver.cc:460] Webserver started at http://127.14.147.194:41865/ using document root <none> and password file <none>
I20260929 22:52:50.817837 29952 fs_manager.cc:359] Metadata directory not provided
I20260929 22:52:50.817885 29952 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 22:52:50.819239 29952 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260929 22:52:50.819917 29967 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 22:52:50.820070 29952 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260929 22:52:50.820150 29952 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790722213368738-14927-0/minicluster-data/ts-1/data,/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790722213368738-14927-0/minicluster-data/ts-1/wal
uuid: "ae4204ae8cb54f4f8f9a085ca5976542"
format_stamp: "Formatted at 2026-09-29 22:52:49 on dist-test-slave-98mk"
server_key: "2e9a6145dda39dc3abf6c3e9ece36713"
server_key_iv: "e70bb44c2b3731c6d82df39796783df4"
server_key_version: "encryptionkey@0"
I20260929 22:52:50.820513 29952 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790722213368738-14927-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790722213368738-14927-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790722213368738-14927-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 22:52:50.855794 29952 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 22:52:50.857924 29952 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 22:52:50.858052 29952 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 22:52:50.858246 29952 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 22:52:50.858565 29952 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 22:52:50.858608 29952 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 22:52:50.858630 29952 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 22:52:50.858641 29952 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 22:52:50.864492 29952 rpc_server.cc:307] RPC server started. Bound to: 127.14.147.194:46603
I20260929 22:52:50.864547 30080 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.147.194:46603 every 8 connection(s)
I20260929 22:52:50.864885 29952 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790722213368738-14927-0/minicluster-data/ts-1/data/info.pb
I20260929 22:52:50.866001 30083 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:52:50.865574 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.147.254:35819 (local address 127.14.147.194:44339)
0929 22:52:50.865738 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0929 22:52:50.865826 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.147.254:35819: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":72,"threads_started":1}
W20260929 22:52:50.866379 30081 heartbeater.cc:644] Failed to heartbeat to 127.14.147.254:35819 (0 consecutive failures): Network error: Failed to ping master at 127.14.147.254:35819: Client connection negotiation failed: client connection to 127.14.147.254:35819: connect: Connection refused (error 111)
I20260929 22:52:50.866642 30083 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:52:50.866558 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.147.254:35819 (local address 127.14.147.194:49851)
0929 22:52:50.866599 (+ 41us) negotiation.cc:107] Waiting for socket to connect
0929 22:52:50.866624 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.147.254:35819: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":26}
I20260929 22:52:50.866879 30083 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:52:50.866807 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.147.254:35819 (local address 127.14.147.194:54259)
0929 22:52:50.866840 (+ 33us) negotiation.cc:107] Waiting for socket to connect
0929 22:52:50.866861 (+ 21us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.147.254:35819: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":21}
W20260929 22:52:50.866993 30081 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260929 22:52:50.867087 14927 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhWOCLk/build/release/bin/kudu as pid 29952
I20260929 22:52:50.867322 14927 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhWOCLk/build/release/bin/kudu
/tmp/dist-test-taskhWOCLk/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790722213368738-14927-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790722213368738-14927-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790722213368738-14927-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790722213368738-14927-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.147.195:41003
--local_ip_for_outbound_sockets=127.14.147.195
--tserver_master_addrs=127.14.147.254:35819
--webserver_port=32777
--webserver_interface=127.14.147.195
--keytab_file=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790722213368738-14927-0/krb5kdc/kudu_127.14.147.195.keytab
--principal=kudu/127.14.147.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.147.212:43149
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790722213368738-14927-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790722213368738-14927-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790722213368738-14927-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 22:52:50.947651 30084 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 22:52:50.947881 30084 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 22:52:50.947908 30084 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 22:52:50.949168 30084 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 22:52:50.949306 30084 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 22:52:50.949388 30084 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.147.195
I20260929 22:52:50.951151 30084 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.147.212:43149
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790722213368738-14927-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790722213368738-14927-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790722213368738-14927-0/krb5kdc/kudu_127.14.147.195.keytab
--principal=kudu/127.14.147.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.147.195:41003
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790722213368738-14927-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.14.147.195
--webserver_port=32777
--tserver_master_addrs=127.14.147.254:35819
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30084
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.147.195
--log_dir=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790722213368738-14927-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 21:58:54 UTC on ccycloud.quasar-xguoti.root.comops.site
build id 4681390
I20260929 22:52:50.951395 30084 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 22:52:50.951658 30084 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 22:52:50.956528 30090 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 22:52:50.956532 30089 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 22:52:50.956528 30092 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 22:52:50.956758 30084 server_base.cc:1034] running on GCE node
I20260929 22:52:50.956955 30084 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 22:52:50.957156 30084 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 22:52:50.958294 30084 hybrid_clock.cc:648] HybridClock initialized: now 1790722370958274 us; error 34 us; skew 500 ppm
Sep 29 22:52:50 dist-test-slave-98mk krb5kdc[28651](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790722370, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.147.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 22:52:50.961222 30084 init.cc:377] Logged in from keytab as kudu/127.14.147.195@KRBTEST.COM (short username kudu)
I20260929 22:52:50.961894 30084 webserver.cc:460] Webserver started at http://127.14.147.195:32777/ using document root <none> and password file <none>
I20260929 22:52:50.962107 30084 fs_manager.cc:359] Metadata directory not provided
I20260929 22:52:50.962152 30084 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 22:52:50.963567 30084 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260929 22:52:50.964145 30099 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 22:52:50.964278 30084 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260929 22:52:50.964360 30084 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790722213368738-14927-0/minicluster-data/ts-2/data,/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790722213368738-14927-0/minicluster-data/ts-2/wal
uuid: "80e8234059154efbbc9417e8b4a71810"
format_stamp: "Formatted at 2026-09-29 22:52:49 on dist-test-slave-98mk"
server_key: "1b6941b1003aef74534e2284c46deca2"
server_key_iv: "4e9b88308cdcc5d800a9607e592d76ae"
server_key_version: "encryptionkey@0"
I20260929 22:52:50.964711 30084 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790722213368738-14927-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790722213368738-14927-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790722213368738-14927-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 22:52:50.984133 30084 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 22:52:50.985985 30084 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 22:52:50.986117 30084 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 22:52:50.986333 30084 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 22:52:50.986629 30084 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 22:52:50.986660 30084 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 22:52:50.986689 30084 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 22:52:50.986712 30084 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 22:52:50.992050 30084 rpc_server.cc:307] RPC server started. Bound to: 127.14.147.195:41003
I20260929 22:52:50.992115 30212 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.147.195:41003 every 8 connection(s)
I20260929 22:52:50.992424 30084 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790722213368738-14927-0/minicluster-data/ts-2/data/info.pb
I20260929 22:52:50.993398 30215 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:52:50.993031 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.147.254:35819 (local address 127.14.147.195:55397)
0929 22:52:50.993173 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0929 22:52:50.993260 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.147.254:35819: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":72,"threads_started":1}
W20260929 22:52:50.993788 30213 heartbeater.cc:644] Failed to heartbeat to 127.14.147.254:35819 (0 consecutive failures): Network error: Failed to ping master at 127.14.147.254:35819: Client connection negotiation failed: client connection to 127.14.147.254:35819: connect: Connection refused (error 111)
I20260929 22:52:50.994021 30215 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:52:50.993941 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.147.254:35819 (local address 127.14.147.195:44237)
0929 22:52:50.993977 (+ 36us) negotiation.cc:107] Waiting for socket to connect
0929 22:52:50.994004 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.147.254:35819: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":22}
I20260929 22:52:50.994257 30215 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:52:50.994190 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.147.254:35819 (local address 127.14.147.195:53585)
0929 22:52:50.994224 (+ 34us) negotiation.cc:107] Waiting for socket to connect
0929 22:52:50.994244 (+ 20us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.147.254:35819: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":24}
W20260929 22:52:50.994359 30213 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260929 22:52:51.002575 14927 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhWOCLk/build/release/bin/kudu as pid 30084
I20260929 22:52:51.769585 30217 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:52:51.769351 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.147.254:35819 (local address 127.14.147.193:43201)
0929 22:52:51.769469 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0929 22:52:51.769508 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.147.254:35819: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":103,"thread_start_us":65,"threads_started":1}
I20260929 22:52:51.867607 30218 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:52:51.867380 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.147.254:35819 (local address 127.14.147.194:38977)
0929 22:52:51.867515 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0929 22:52:51.867553 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.147.254:35819: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":83,"threads_started":1}
I20260929 22:52:51.994897 30219 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:52:51.994692 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.147.254:35819 (local address 127.14.147.195:47919)
0929 22:52:51.994805 (+ 113us) negotiation.cc:107] Waiting for socket to connect
0929 22:52:51.994843 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.147.254:35819: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":95,"thread_start_us":61,"threads_started":1}
I20260929 22:52:52.272770 30220 raft_consensus.cc:482] T 984df8b1c28e43b2882f3802f7ff2238 P 43f3886bb1b44e4aabb3e8f71ff6f3b5 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 22:52:52.272908 30220 raft_consensus.cc:504] T 984df8b1c28e43b2882f3802f7ff2238 P 43f3886bb1b44e4aabb3e8f71ff6f3b5 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae4204ae8cb54f4f8f9a085ca5976542" member_type: VOTER last_known_addr { host: "127.14.147.194" port: 46603 } } peers { permanent_uuid: "80e8234059154efbbc9417e8b4a71810" member_type: VOTER last_known_addr { host: "127.14.147.195" port: 41003 } } peers { permanent_uuid: "43f3886bb1b44e4aabb3e8f71ff6f3b5" member_type: VOTER last_known_addr { host: "127.14.147.193" port: 46859 } }
I20260929 22:52:52.273254 30220 leader_election.cc:288] T 984df8b1c28e43b2882f3802f7ff2238 P 43f3886bb1b44e4aabb3e8f71ff6f3b5 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ae4204ae8cb54f4f8f9a085ca5976542 (127.14.147.194:46603), 80e8234059154efbbc9417e8b4a71810 (127.14.147.195:41003)
Sep 29 22:52:52 dist-test-slave-98mk krb5kdc[28651](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790722370, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.147.193@KRBTEST.COM for kudu/127.14.147.195@KRBTEST.COM
Sep 29 22:52:52 dist-test-slave-98mk krb5kdc[28651](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790722370, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.147.193@KRBTEST.COM for kudu/127.14.147.194@KRBTEST.COM
I20260929 22:52:52.284435 30223 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:52:52.273566 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.147.193:47427 (local address 127.14.147.194:46603)
0929 22:52:52.273732 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0929 22:52:52.273744 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0929 22:52:52.273775 (+ 31us) server_negotiation.cc:407] Connection header received
0929 22:52:52.274223 (+ 448us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 22:52:52.274226 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 22:52:52.274269 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 22:52:52.274412 (+ 143us) server_negotiation.cc:226] Negotiated authn=SASL
0929 22:52:52.275301 (+ 889us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:52:52.276316 (+ 1015us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:52:52.277255 (+ 939us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:52:52.277661 (+ 406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:52:52.277756 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 22:52:52.280737 (+ 2981us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 22:52:52.280742 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 22:52:52.280742 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 22:52:52.280754 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0929 22:52:52.282715 (+ 1961us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 22:52:52.283140 (+ 425us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 22:52:52.283143 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 22:52:52.283144 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 22:52:52.283238 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 22:52:52.283546 (+ 308us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 22:52:52.283547 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 22:52:52.283547 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 22:52:52.283812 (+ 265us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 22:52:52.283888 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0929 22:52:52.284176 (+ 288us) server_negotiation.cc:299] Negotiation successful
0929 22:52:52.284317 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":131,"thread_start_us":47,"threads_started":1}
I20260929 22:52:52.284432 30222 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:52:52.273489 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.147.194:46603 (local address 127.14.147.193:47427)
0929 22:52:52.273605 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0929 22:52:52.273616 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0929 22:52:52.273684 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 22:52:52.274458 (+ 774us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 22:52:52.274461 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 22:52:52.274825 (+ 364us) client_negotiation.cc:189] Negotiated authn=SASL
0929 22:52:52.275192 (+ 367us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 22:52:52.275194 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 22:52:52.276406 (+ 1212us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 22:52:52.276407 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 22:52:52.277171 (+ 764us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 22:52:52.277173 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 22:52:52.277746 (+ 573us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 22:52:52.277748 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 22:52:52.278083 (+ 335us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 22:52:52.278646 (+ 563us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 22:52:52.278650 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0929 22:52:52.280624 (+ 1974us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 22:52:52.282792 (+ 2168us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 22:52:52.282802 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 22:52:52.282805 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 22:52:52.283043 (+ 238us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 22:52:52.283329 (+ 286us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 22:52:52.283330 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 22:52:52.283331 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 22:52:52.283484 (+ 153us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 22:52:52.283879 (+ 395us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 22:52:52.283886 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 22:52:52.284021 (+ 135us) client_negotiation.cc:769] Sending connection context
0929 22:52:52.284109 (+ 88us) client_negotiation.cc:240] Negotiation successful
0929 22:52:52.284323 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":102,"mutex_wait_us":15,"thread_start_us":57,"threads_started":1}
I20260929 22:52:52.284610 30224 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:52:52.273661 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.147.193:46323 (local address 127.14.147.195:41003)
0929 22:52:52.273793 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0929 22:52:52.273803 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 22:52:52.273828 (+ 25us) server_negotiation.cc:407] Connection header received
0929 22:52:52.274230 (+ 402us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 22:52:52.274232 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 22:52:52.274275 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 22:52:52.274378 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0929 22:52:52.275175 (+ 797us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:52:52.276228 (+ 1053us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:52:52.277255 (+ 1027us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:52:52.277658 (+ 403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:52:52.277757 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 22:52:52.280419 (+ 2662us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 22:52:52.280424 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 22:52:52.280424 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 22:52:52.280704 (+ 280us) server_negotiation.cc:922] calling sasl_server_start()
0929 22:52:52.282712 (+ 2008us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 22:52:52.283340 (+ 628us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 22:52:52.283349 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 22:52:52.283350 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 22:52:52.283471 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 22:52:52.283636 (+ 165us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 22:52:52.283637 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 22:52:52.283637 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 22:52:52.283870 (+ 233us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 22:52:52.283938 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0929 22:52:52.284174 (+ 236us) server_negotiation.cc:299] Negotiation successful
0929 22:52:52.284305 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":98,"thread_start_us":64,"threads_started":1}
I20260929 22:52:52.284907 30221 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:52:52.273458 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.147.195:41003 (local address 127.14.147.193:46323)
0929 22:52:52.273576 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0929 22:52:52.273588 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0929 22:52:52.273651 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 22:52:52.274485 (+ 834us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 22:52:52.274486 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 22:52:52.274827 (+ 341us) client_negotiation.cc:189] Negotiated authn=SASL
0929 22:52:52.275058 (+ 231us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 22:52:52.275064 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 22:52:52.276320 (+ 1256us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 22:52:52.276321 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 22:52:52.277172 (+ 851us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 22:52:52.277174 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 22:52:52.277744 (+ 570us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 22:52:52.277745 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 22:52:52.278033 (+ 288us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 22:52:52.278596 (+ 563us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 22:52:52.278604 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0929 22:52:52.280300 (+ 1696us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 22:52:52.283113 (+ 2813us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 22:52:52.283114 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 22:52:52.283114 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 22:52:52.283262 (+ 148us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 22:52:52.283531 (+ 269us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 22:52:52.283532 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 22:52:52.283532 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 22:52:52.283584 (+ 52us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 22:52:52.283934 (+ 350us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 22:52:52.283935 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 22:52:52.284021 (+ 86us) client_negotiation.cc:769] Sending connection context
0929 22:52:52.284109 (+ 88us) client_negotiation.cc:240] Negotiation successful
0929 22:52:52.284789 (+ 680us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":101,"mutex_wait_us":158,"thread_start_us":40,"threads_started":1}
W20260929 22:52:52.285696 29834 leader_election.cc:341] T 984df8b1c28e43b2882f3802f7ff2238 P 43f3886bb1b44e4aabb3e8f71ff6f3b5 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer ae4204ae8cb54f4f8f9a085ca5976542 (127.14.147.194:46603): Not found: Tablet not found: 984df8b1c28e43b2882f3802f7ff2238
W20260929 22:52:52.286162 29833 leader_election.cc:341] T 984df8b1c28e43b2882f3802f7ff2238 P 43f3886bb1b44e4aabb3e8f71ff6f3b5 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 80e8234059154efbbc9417e8b4a71810 (127.14.147.195:41003): Not found: Tablet not found: 984df8b1c28e43b2882f3802f7ff2238
I20260929 22:52:52.286244 29833 leader_election.cc:302] T 984df8b1c28e43b2882f3802f7ff2238 P 43f3886bb1b44e4aabb3e8f71ff6f3b5 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 43f3886bb1b44e4aabb3e8f71ff6f3b5; no voters: 80e8234059154efbbc9417e8b4a71810, ae4204ae8cb54f4f8f9a085ca5976542
I20260929 22:52:52.286376 30220 raft_consensus.cc:2726] T 984df8b1c28e43b2882f3802f7ff2238 P 43f3886bb1b44e4aabb3e8f71ff6f3b5 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260929 22:52:52.770159 30221 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:52:52.770014 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.147.254:35819 (local address 127.14.147.193:45925)
0929 22:52:52.770070 (+ 56us) negotiation.cc:107] Waiting for socket to connect
0929 22:52:52.770116 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.147.254:35819: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
I20260929 22:52:52.868185 30225 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:52:52.867964 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.147.254:35819 (local address 127.14.147.194:51937)
0929 22:52:52.868095 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0929 22:52:52.868139 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.147.254:35819: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":60,"threads_started":1}
I20260929 22:52:52.995645 30226 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:52:52.995445 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.147.254:35819 (local address 127.14.147.195:57761)
0929 22:52:52.995561 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0929 22:52:52.995603 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.147.254:35819: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":96,"thread_start_us":58,"threads_started":1}
I20260929 22:52:53.770859 30227 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:52:53.770629 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.147.254:35819 (local address 127.14.147.193:48641)
0929 22:52:53.770770 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0929 22:52:53.770806 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.147.254:35819: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":64,"threads_started":1}
I20260929 22:52:53.868914 30228 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:52:53.868663 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.147.254:35819 (local address 127.14.147.194:51169)
0929 22:52:53.868807 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0929 22:52:53.868850 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.147.254:35819: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":60,"threads_started":1}
I20260929 22:52:53.996246 30229 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:52:53.996047 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.147.254:35819 (local address 127.14.147.195:53583)
0929 22:52:53.996166 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0929 22:52:53.996203 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.147.254:35819: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":103,"thread_start_us":61,"threads_started":1}
I20260929 22:52:54.312914 30230 raft_consensus.cc:482] T 984df8b1c28e43b2882f3802f7ff2238 P 43f3886bb1b44e4aabb3e8f71ff6f3b5 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 22:52:54.313050 30230 raft_consensus.cc:504] T 984df8b1c28e43b2882f3802f7ff2238 P 43f3886bb1b44e4aabb3e8f71ff6f3b5 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae4204ae8cb54f4f8f9a085ca5976542" member_type: VOTER last_known_addr { host: "127.14.147.194" port: 46603 } } peers { permanent_uuid: "80e8234059154efbbc9417e8b4a71810" member_type: VOTER last_known_addr { host: "127.14.147.195" port: 41003 } } peers { permanent_uuid: "43f3886bb1b44e4aabb3e8f71ff6f3b5" member_type: VOTER last_known_addr { host: "127.14.147.193" port: 46859 } }
I20260929 22:52:54.313325 30230 leader_election.cc:288] T 984df8b1c28e43b2882f3802f7ff2238 P 43f3886bb1b44e4aabb3e8f71ff6f3b5 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ae4204ae8cb54f4f8f9a085ca5976542 (127.14.147.194:46603), 80e8234059154efbbc9417e8b4a71810 (127.14.147.195:41003)
W20260929 22:52:54.313611 29833 leader_election.cc:341] T 984df8b1c28e43b2882f3802f7ff2238 P 43f3886bb1b44e4aabb3e8f71ff6f3b5 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 80e8234059154efbbc9417e8b4a71810 (127.14.147.195:41003): Not found: Tablet not found: 984df8b1c28e43b2882f3802f7ff2238
W20260929 22:52:54.313671 29834 leader_election.cc:341] T 984df8b1c28e43b2882f3802f7ff2238 P 43f3886bb1b44e4aabb3e8f71ff6f3b5 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer ae4204ae8cb54f4f8f9a085ca5976542 (127.14.147.194:46603): Not found: Tablet not found: 984df8b1c28e43b2882f3802f7ff2238
I20260929 22:52:54.313771 29834 leader_election.cc:302] T 984df8b1c28e43b2882f3802f7ff2238 P 43f3886bb1b44e4aabb3e8f71ff6f3b5 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 43f3886bb1b44e4aabb3e8f71ff6f3b5; no voters: 80e8234059154efbbc9417e8b4a71810, ae4204ae8cb54f4f8f9a085ca5976542
I20260929 22:52:54.313846 30230 raft_consensus.cc:2726] T 984df8b1c28e43b2882f3802f7ff2238 P 43f3886bb1b44e4aabb3e8f71ff6f3b5 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260929 22:52:54.771626 30231 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:52:54.771337 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.147.254:35819 (local address 127.14.147.193:43263)
0929 22:52:54.771529 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0929 22:52:54.771572 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.147.254:35819: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":172,"thread_start_us":138,"threads_started":1}
I20260929 22:52:54.869629 30232 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:52:54.869389 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.147.254:35819 (local address 127.14.147.194:36453)
0929 22:52:54.869539 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0929 22:52:54.869581 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.147.254:35819: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":76,"threads_started":1}
I20260929 22:52:54.996987 30233 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:52:54.996765 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.147.254:35819 (local address 127.14.147.195:53807)
0929 22:52:54.996895 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0929 22:52:54.996938 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.147.254:35819: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":70,"threads_started":1}
I20260929 22:52:55.772353 30234 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:52:55.772112 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.147.254:35819 (local address 127.14.147.193:51711)
0929 22:52:55.772252 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0929 22:52:55.772301 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.147.254:35819: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":61,"threads_started":1}
I20260929 22:52:55.870456 30235 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:52:55.870210 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.147.254:35819 (local address 127.14.147.194:55743)
0929 22:52:55.870362 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0929 22:52:55.870408 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.147.254:35819: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":67,"threads_started":1}
I20260929 22:52:55.997687 30236 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:52:55.997496 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.147.254:35819 (local address 127.14.147.195:45457)
0929 22:52:55.997605 (+ 109us) negotiation.cc:107] Waiting for socket to connect
0929 22:52:55.997647 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.147.254:35819: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":94,"thread_start_us":59,"threads_started":1}
I20260929 22:52:56.773085 30237 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:52:56.772828 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.147.254:35819 (local address 127.14.147.193:38265)
0929 22:52:56.772986 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0929 22:52:56.773036 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.147.254:35819: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":135,"thread_start_us":63,"threads_started":1}
I20260929 22:52:56.871129 30238 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:52:56.870905 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.147.254:35819 (local address 127.14.147.194:49357)
0929 22:52:56.871044 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0929 22:52:56.871080 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.147.254:35819: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":71,"threads_started":1}
I20260929 22:52:56.871544 30239 raft_consensus.cc:482] T 984df8b1c28e43b2882f3802f7ff2238 P 43f3886bb1b44e4aabb3e8f71ff6f3b5 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 22:52:56.871615 30239 raft_consensus.cc:504] T 984df8b1c28e43b2882f3802f7ff2238 P 43f3886bb1b44e4aabb3e8f71ff6f3b5 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae4204ae8cb54f4f8f9a085ca5976542" member_type: VOTER last_known_addr { host: "127.14.147.194" port: 46603 } } peers { permanent_uuid: "80e8234059154efbbc9417e8b4a71810" member_type: VOTER last_known_addr { host: "127.14.147.195" port: 41003 } } peers { permanent_uuid: "43f3886bb1b44e4aabb3e8f71ff6f3b5" member_type: VOTER last_known_addr { host: "127.14.147.193" port: 46859 } }
I20260929 22:52:56.871789 30239 leader_election.cc:288] T 984df8b1c28e43b2882f3802f7ff2238 P 43f3886bb1b44e4aabb3e8f71ff6f3b5 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ae4204ae8cb54f4f8f9a085ca5976542 (127.14.147.194:46603), 80e8234059154efbbc9417e8b4a71810 (127.14.147.195:41003)
W20260929 22:52:56.872093 29834 leader_election.cc:341] T 984df8b1c28e43b2882f3802f7ff2238 P 43f3886bb1b44e4aabb3e8f71ff6f3b5 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer ae4204ae8cb54f4f8f9a085ca5976542 (127.14.147.194:46603): Not found: Tablet not found: 984df8b1c28e43b2882f3802f7ff2238
W20260929 22:52:56.872166 29833 leader_election.cc:341] T 984df8b1c28e43b2882f3802f7ff2238 P 43f3886bb1b44e4aabb3e8f71ff6f3b5 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 80e8234059154efbbc9417e8b4a71810 (127.14.147.195:41003): Not found: Tablet not found: 984df8b1c28e43b2882f3802f7ff2238
I20260929 22:52:56.872216 29833 leader_election.cc:302] T 984df8b1c28e43b2882f3802f7ff2238 P 43f3886bb1b44e4aabb3e8f71ff6f3b5 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 43f3886bb1b44e4aabb3e8f71ff6f3b5; no voters: 80e8234059154efbbc9417e8b4a71810, ae4204ae8cb54f4f8f9a085ca5976542
I20260929 22:52:56.872292 30239 raft_consensus.cc:2726] T 984df8b1c28e43b2882f3802f7ff2238 P 43f3886bb1b44e4aabb3e8f71ff6f3b5 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260929 22:52:56.998317 30240 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:52:56.998118 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.147.254:35819 (local address 127.14.147.195:51729)
0929 22:52:56.998242 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0929 22:52:56.998271 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.147.254:35819: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":60,"threads_started":1}
I20260929 22:52:57.773782 30241 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:52:57.773572 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.147.254:35819 (local address 127.14.147.193:48285)
0929 22:52:57.773694 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0929 22:52:57.773733 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.147.254:35819: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":106,"thread_start_us":64,"threads_started":1}
I20260929 22:52:57.871874 30242 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:52:57.871641 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.147.254:35819 (local address 127.14.147.194:55441)
0929 22:52:57.871785 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0929 22:52:57.871828 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.147.254:35819: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":61,"threads_started":1}
I20260929 22:52:57.999017 30243 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:52:57.998789 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.147.254:35819 (local address 127.14.147.195:48913)
0929 22:52:57.998925 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0929 22:52:57.998967 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.147.254:35819: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":67,"threads_started":1}
I20260929 22:52:58.774525 30244 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:52:58.774285 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.147.254:35819 (local address 127.14.147.193:58741)
0929 22:52:58.774431 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0929 22:52:58.774477 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.147.254:35819: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":81,"threads_started":1}
I20260929 22:52:58.872536 30245 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:52:58.872295 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.147.254:35819 (local address 127.14.147.194:42447)
0929 22:52:58.872428 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0929 22:52:58.872473 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.147.254:35819: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":66,"threads_started":1}
I20260929 22:52:58.999811 30246 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:52:58.999578 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.147.254:35819 (local address 127.14.147.195:48925)
0929 22:52:58.999722 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0929 22:52:58.999764 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.147.254:35819: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":65,"threads_started":1}
I20260929 22:52:59.775261 30247 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:52:59.774999 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.147.254:35819 (local address 127.14.147.193:34243)
0929 22:52:59.775168 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0929 22:52:59.775213 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.147.254:35819: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":147,"thread_start_us":79,"threads_started":1}
I20260929 22:52:59.873310 30248 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:52:59.873081 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.147.254:35819 (local address 127.14.147.194:50657)
0929 22:52:59.873221 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0929 22:52:59.873260 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.147.254:35819: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":77,"threads_started":1}
I20260929 22:53:00.000646 30249 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:53:00.000257 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.147.254:35819 (local address 127.14.147.195:46141)
0929 22:53:00.000488 (+ 231us) negotiation.cc:107] Waiting for socket to connect
0929 22:53:00.000598 (+ 110us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.147.254:35819: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":193,"thread_start_us":79,"threads_started":1}
I20260929 22:53:00.775931 30250 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:53:00.775743 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.147.254:35819 (local address 127.14.147.193:43995)
0929 22:53:00.775856 (+ 113us) negotiation.cc:107] Waiting for socket to connect
0929 22:53:00.775885 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.147.254:35819: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":99,"thread_start_us":60,"threads_started":1}
I20260929 22:53:00.870920 30251 raft_consensus.cc:482] T 984df8b1c28e43b2882f3802f7ff2238 P 43f3886bb1b44e4aabb3e8f71ff6f3b5 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 22:53:00.871019 30251 raft_consensus.cc:504] T 984df8b1c28e43b2882f3802f7ff2238 P 43f3886bb1b44e4aabb3e8f71ff6f3b5 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae4204ae8cb54f4f8f9a085ca5976542" member_type: VOTER last_known_addr { host: "127.14.147.194" port: 46603 } } peers { permanent_uuid: "80e8234059154efbbc9417e8b4a71810" member_type: VOTER last_known_addr { host: "127.14.147.195" port: 41003 } } peers { permanent_uuid: "43f3886bb1b44e4aabb3e8f71ff6f3b5" member_type: VOTER last_known_addr { host: "127.14.147.193" port: 46859 } }
I20260929 22:53:00.871191 30251 leader_election.cc:288] T 984df8b1c28e43b2882f3802f7ff2238 P 43f3886bb1b44e4aabb3e8f71ff6f3b5 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ae4204ae8cb54f4f8f9a085ca5976542 (127.14.147.194:46603), 80e8234059154efbbc9417e8b4a71810 (127.14.147.195:41003)
W20260929 22:53:00.871614 29834 leader_election.cc:341] T 984df8b1c28e43b2882f3802f7ff2238 P 43f3886bb1b44e4aabb3e8f71ff6f3b5 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer ae4204ae8cb54f4f8f9a085ca5976542 (127.14.147.194:46603): Not found: Tablet not found: 984df8b1c28e43b2882f3802f7ff2238
W20260929 22:53:00.871688 29833 leader_election.cc:341] T 984df8b1c28e43b2882f3802f7ff2238 P 43f3886bb1b44e4aabb3e8f71ff6f3b5 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 80e8234059154efbbc9417e8b4a71810 (127.14.147.195:41003): Not found: Tablet not found: 984df8b1c28e43b2882f3802f7ff2238
I20260929 22:53:00.871755 29833 leader_election.cc:302] T 984df8b1c28e43b2882f3802f7ff2238 P 43f3886bb1b44e4aabb3e8f71ff6f3b5 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 43f3886bb1b44e4aabb3e8f71ff6f3b5; no voters: 80e8234059154efbbc9417e8b4a71810, ae4204ae8cb54f4f8f9a085ca5976542
I20260929 22:53:00.871827 30251 raft_consensus.cc:2726] T 984df8b1c28e43b2882f3802f7ff2238 P 43f3886bb1b44e4aabb3e8f71ff6f3b5 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260929 22:53:00.874010 30252 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:53:00.873804 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.147.254:35819 (local address 127.14.147.194:58765)
0929 22:53:00.873916 (+ 112us) negotiation.cc:107] Waiting for socket to connect
0929 22:53:00.873963 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.147.254:35819: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":95,"thread_start_us":61,"threads_started":1}
I20260929 22:53:01.001354 30253 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:53:01.001143 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.147.254:35819 (local address 127.14.147.195:44115)
0929 22:53:01.001266 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0929 22:53:01.001306 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.147.254:35819: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":104,"thread_start_us":60,"threads_started":1}
I20260929 22:53:01.776593 30254 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:53:01.776362 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.147.254:35819 (local address 127.14.147.193:59905)
0929 22:53:01.776504 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0929 22:53:01.776545 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.147.254:35819: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":72,"threads_started":1}
I20260929 22:53:01.874738 30255 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:53:01.874522 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.147.254:35819 (local address 127.14.147.194:34355)
0929 22:53:01.874643 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0929 22:53:01.874690 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.147.254:35819: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":103,"thread_start_us":64,"threads_started":1}
I20260929 22:53:02.002072 30256 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:53:02.001795 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.147.254:35819 (local address 127.14.147.195:56603)
0929 22:53:02.001972 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0929 22:53:02.002023 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.147.254:35819: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":149,"thread_start_us":69,"threads_started":1}
I20260929 22:53:02.777280 30257 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:53:02.777078 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.147.254:35819 (local address 127.14.147.193:58949)
0929 22:53:02.777193 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0929 22:53:02.777232 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.147.254:35819: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":98,"thread_start_us":60,"threads_started":1}
I20260929 22:53:02.875489 30258 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:53:02.875212 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.147.254:35819 (local address 127.14.147.194:53467)
0929 22:53:02.875361 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0929 22:53:02.875406 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.147.254:35819: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":65,"threads_started":1}
I20260929 22:53:03.002928 30259 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:53:03.002682 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.147.254:35819 (local address 127.14.147.195:44797)
0929 22:53:03.002837 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0929 22:53:03.002879 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.147.254:35819: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":136,"thread_start_us":76,"threads_started":1}
I20260929 22:53:03.187635 30260 raft_consensus.cc:482] T 984df8b1c28e43b2882f3802f7ff2238 P 43f3886bb1b44e4aabb3e8f71ff6f3b5 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 22:53:03.187784 30260 raft_consensus.cc:504] T 984df8b1c28e43b2882f3802f7ff2238 P 43f3886bb1b44e4aabb3e8f71ff6f3b5 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae4204ae8cb54f4f8f9a085ca5976542" member_type: VOTER last_known_addr { host: "127.14.147.194" port: 46603 } } peers { permanent_uuid: "80e8234059154efbbc9417e8b4a71810" member_type: VOTER last_known_addr { host: "127.14.147.195" port: 41003 } } peers { permanent_uuid: "43f3886bb1b44e4aabb3e8f71ff6f3b5" member_type: VOTER last_known_addr { host: "127.14.147.193" port: 46859 } }
I20260929 22:53:03.187994 30260 leader_election.cc:288] T 984df8b1c28e43b2882f3802f7ff2238 P 43f3886bb1b44e4aabb3e8f71ff6f3b5 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ae4204ae8cb54f4f8f9a085ca5976542 (127.14.147.194:46603), 80e8234059154efbbc9417e8b4a71810 (127.14.147.195:41003)
W20260929 22:53:03.188369 29834 leader_election.cc:341] T 984df8b1c28e43b2882f3802f7ff2238 P 43f3886bb1b44e4aabb3e8f71ff6f3b5 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer ae4204ae8cb54f4f8f9a085ca5976542 (127.14.147.194:46603): Not found: Tablet not found: 984df8b1c28e43b2882f3802f7ff2238
W20260929 22:53:03.188474 29833 leader_election.cc:341] T 984df8b1c28e43b2882f3802f7ff2238 P 43f3886bb1b44e4aabb3e8f71ff6f3b5 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 80e8234059154efbbc9417e8b4a71810 (127.14.147.195:41003): Not found: Tablet not found: 984df8b1c28e43b2882f3802f7ff2238
I20260929 22:53:03.188539 29833 leader_election.cc:302] T 984df8b1c28e43b2882f3802f7ff2238 P 43f3886bb1b44e4aabb3e8f71ff6f3b5 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 43f3886bb1b44e4aabb3e8f71ff6f3b5; no voters: 80e8234059154efbbc9417e8b4a71810, ae4204ae8cb54f4f8f9a085ca5976542
I20260929 22:53:03.188621 30260 raft_consensus.cc:2726] T 984df8b1c28e43b2882f3802f7ff2238 P 43f3886bb1b44e4aabb3e8f71ff6f3b5 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260929 22:53:03.778026 30261 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:53:03.777786 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.147.254:35819 (local address 127.14.147.193:39903)
0929 22:53:03.777931 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0929 22:53:03.777975 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.147.254:35819: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":84,"threads_started":1}
I20260929 22:53:03.876148 30262 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:53:03.875952 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.147.254:35819 (local address 127.14.147.194:33037)
0929 22:53:03.876068 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0929 22:53:03.876102 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.147.254:35819: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":102,"thread_start_us":65,"threads_started":1}
I20260929 22:53:04.003685 30263 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:53:04.003477 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.147.254:35819 (local address 127.14.147.195:32909)
0929 22:53:04.003610 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0929 22:53:04.003639 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.147.254:35819: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":69,"threads_started":1}
I20260929 22:53:04.778738 30264 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:53:04.778474 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.147.254:35819 (local address 127.14.147.193:44407)
0929 22:53:04.778643 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0929 22:53:04.778689 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.147.254:35819: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":149,"thread_start_us":80,"threads_started":1}
I20260929 22:53:04.876909 30265 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:53:04.876629 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.147.254:35819 (local address 127.14.147.194:35137)
0929 22:53:04.876810 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0929 22:53:04.876859 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.147.254:35819: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":160,"thread_start_us":89,"threads_started":1}
I20260929 22:53:05.004469 30266 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:53:05.004123 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.147.254:35819 (local address 127.14.147.195:52805)
0929 22:53:05.004324 (+ 201us) negotiation.cc:107] Waiting for socket to connect
0929 22:53:05.004400 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.147.254:35819: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":166,"thread_start_us":68,"threads_started":1}
I20260929 22:53:05.779522 30267 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:53:05.779289 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.147.254:35819 (local address 127.14.147.193:33587)
0929 22:53:05.779431 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0929 22:53:05.779471 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.147.254:35819: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":103,"thread_start_us":67,"threads_started":1}
I20260929 22:53:05.877889 30268 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:53:05.877655 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.147.254:35819 (local address 127.14.147.194:46073)
0929 22:53:05.877794 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0929 22:53:05.877835 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.147.254:35819: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":73,"threads_started":1}
I20260929 22:53:06.005367 30269 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:53:06.005113 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.147.254:35819 (local address 127.14.147.195:47293)
0929 22:53:06.005266 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0929 22:53:06.005321 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.147.254:35819: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":57,"threads_started":1}
I20260929 22:53:06.010226 14927 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhWOCLk/build/release/bin/kudu with pid 29788
I20260929 22:53:06.014359 14927 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhWOCLk/build/release/bin/kudu with pid 29952
I20260929 22:53:06.018139 14927 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhWOCLk/build/release/bin/kudu with pid 30084
2026-09-29T22:53:06Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedIPKICert (18336 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedTSK
Loading random data
Initializing database '/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790722213368738-14927-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 22:53:06 dist-test-slave-98mk krb5kdc[30272](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 22:53:06 dist-test-slave-98mk krb5kdc[30272](info): set up 2 sockets
Sep 29 22:53:06 dist-test-slave-98mk krb5kdc[30272](info): commencing operation
krb5kdc: starting...
W20260929 22:53:06.983996 14927 mini_kdc.cc:121] Time spent starting KDC: real 0.947s user 0.005s sys 0.000s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790722213368738-14927-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790722213368738-14927-0/krb5kdc/test-user.keytab.
Sep 29 22:53:07 dist-test-slave-98mk krb5kdc[30272](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790722387, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T22:53:07Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T22:53:07Z Disabled control of system clock
WARNING: no policy specified for kudu/127.14.147.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.147.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.147.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790722213368738-14927-0/krb5kdc/kudu_127.14.147.254.keytab.
Entry for principal kudu/127.14.147.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790722213368738-14927-0/krb5kdc/kudu_127.14.147.254.keytab.
I20260929 22:53:07.100347 14927 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhWOCLk/build/release/bin/kudu
/tmp/dist-test-taskhWOCLk/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790722213368738-14927-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790722213368738-14927-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790722213368738-14927-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790722213368738-14927-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.147.254:35531
--webserver_interface=127.14.147.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790722213368738-14927-0/krb5kdc/kudu_127.14.147.254.keytab
--principal=kudu/127.14.147.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.147.212:41357
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.147.254:35531
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790722213368738-14927-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790722213368738-14927-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790722213368738-14927-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 22:53:07.180536 30285 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 22:53:07.180763 30285 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 22:53:07.180796 30285 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 22:53:07.182041 30285 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260929 22:53:07.182091 30285 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260929 22:53:07.182107 30285 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260929 22:53:07.182121 30285 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 22:53:07.182138 30285 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 22:53:07.182154 30285 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 22:53:07.182173 30285 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 22:53:07.182191 30285 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 22:53:07.183811 30285 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.147.212:41357
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790722213368738-14927-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790722213368738-14927-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.14.147.254:35531
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790722213368738-14927-0/krb5kdc/kudu_127.14.147.254.keytab
--principal=kudu/127.14.147.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.147.254:35531
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790722213368738-14927-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.14.147.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30285
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790722213368738-14927-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 21:58:54 UTC on ccycloud.quasar-xguoti.root.comops.site
build id 4681390
I20260929 22:53:07.184082 30285 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 22:53:07.184300 30285 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 22:53:07.189285 30291 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 22:53:07.189386 30290 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 22:53:07.189285 30293 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 22:53:07.189483 30285 server_base.cc:1034] running on GCE node
I20260929 22:53:07.189761 30285 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 22:53:07.190032 30285 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 22:53:07.191210 30285 hybrid_clock.cc:648] HybridClock initialized: now 1790722387191189 us; error 38 us; skew 500 ppm
Sep 29 22:53:07 dist-test-slave-98mk krb5kdc[30272](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790722387, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.147.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 22:53:07.193655 30285 init.cc:377] Logged in from keytab as kudu/127.14.147.254@KRBTEST.COM (short username kudu)
I20260929 22:53:07.194115 30285 webserver.cc:460] Webserver started at http://127.14.147.254:46623/ using document root <none> and password file <none>
I20260929 22:53:07.194334 30285 fs_manager.cc:359] Metadata directory not provided
I20260929 22:53:07.194386 30285 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 22:53:07.194505 30285 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 22:53:07.195609 30285 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790722213368738-14927-0/minicluster-data/master-0/data/instance:
uuid: "34cef3aa11664e048d242d3feef47476"
format_stamp: "Formatted at 2026-09-29 22:53:07 on dist-test-slave-98mk"
server_key: "19382e3e92633e92632c2d8bb0006584"
server_key_iv: "067b6736fb0621bad8f856c90cab1be8"
server_key_version: "encryptionkey@0"
I20260929 22:53:07.195937 30285 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790722213368738-14927-0/minicluster-data/master-0/wal/instance:
uuid: "34cef3aa11664e048d242d3feef47476"
format_stamp: "Formatted at 2026-09-29 22:53:07 on dist-test-slave-98mk"
server_key: "19382e3e92633e92632c2d8bb0006584"
server_key_iv: "067b6736fb0621bad8f856c90cab1be8"
server_key_version: "encryptionkey@0"
I20260929 22:53:07.197258 30285 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260929 22:53:07.198136 30300 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 22:53:07.198326 30285 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260929 22:53:07.198412 30285 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790722213368738-14927-0/minicluster-data/master-0/data,/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790722213368738-14927-0/minicluster-data/master-0/wal
uuid: "34cef3aa11664e048d242d3feef47476"
format_stamp: "Formatted at 2026-09-29 22:53:07 on dist-test-slave-98mk"
server_key: "19382e3e92633e92632c2d8bb0006584"
server_key_iv: "067b6736fb0621bad8f856c90cab1be8"
server_key_version: "encryptionkey@0"
I20260929 22:53:07.198472 30285 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790722213368738-14927-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790722213368738-14927-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790722213368738-14927-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 22:53:07.209810 30285 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 22:53:07.212028 30285 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 22:53:07.212265 30285 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 22:53:07.216194 30285 rpc_server.cc:307] RPC server started. Bound to: 127.14.147.254:35531
I20260929 22:53:07.216254 30352 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.147.254:35531 every 8 connection(s)
I20260929 22:53:07.216619 30285 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790722213368738-14927-0/minicluster-data/master-0/data/info.pb
I20260929 22:53:07.217522 30353 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 22:53:07.220441 30353 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 34cef3aa11664e048d242d3feef47476: Bootstrap starting.
I20260929 22:53:07.221135 30353 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 34cef3aa11664e048d242d3feef47476: Neither blocks nor log segments found. Creating new log.
I20260929 22:53:07.221427 30353 log.cc:824] T 00000000000000000000000000000000 P 34cef3aa11664e048d242d3feef47476: Log is configured to *not* fsync() on all Append() calls
I20260929 22:53:07.222414 30353 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 34cef3aa11664e048d242d3feef47476: No bootstrap required, opened a new log
I20260929 22:53:07.224180 30353 raft_consensus.cc:348] T 00000000000000000000000000000000 P 34cef3aa11664e048d242d3feef47476 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "34cef3aa11664e048d242d3feef47476" member_type: VOTER last_known_addr { host: "127.14.147.254" port: 35531 } }
I20260929 22:53:07.224354 30353 raft_consensus.cc:374] T 00000000000000000000000000000000 P 34cef3aa11664e048d242d3feef47476 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 22:53:07.224391 30353 raft_consensus.cc:729] T 00000000000000000000000000000000 P 34cef3aa11664e048d242d3feef47476 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 34cef3aa11664e048d242d3feef47476, State: Initialized, Role: FOLLOWER
I20260929 22:53:07.224539 30353 consensus_queue.cc:260] T 00000000000000000000000000000000 P 34cef3aa11664e048d242d3feef47476 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "34cef3aa11664e048d242d3feef47476" member_type: VOTER last_known_addr { host: "127.14.147.254" port: 35531 } }
I20260929 22:53:07.224619 30353 raft_consensus.cc:388] T 00000000000000000000000000000000 P 34cef3aa11664e048d242d3feef47476 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 22:53:07.224653 30353 raft_consensus.cc:482] T 00000000000000000000000000000000 P 34cef3aa11664e048d242d3feef47476 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 22:53:07.224701 30353 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 34cef3aa11664e048d242d3feef47476 [term 0 FOLLOWER]: Advancing to term 1
I20260929 22:53:07.225317 14927 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhWOCLk/build/release/bin/kudu as pid 30285
I20260929 22:53:07.225389 30353 raft_consensus.cc:504] T 00000000000000000000000000000000 P 34cef3aa11664e048d242d3feef47476 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "34cef3aa11664e048d242d3feef47476" member_type: VOTER last_known_addr { host: "127.14.147.254" port: 35531 } }
I20260929 22:53:07.225433 14927 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790722213368738-14927-0/minicluster-data/master-0/wal/instance
I20260929 22:53:07.225492 30353 leader_election.cc:302] T 00000000000000000000000000000000 P 34cef3aa11664e048d242d3feef47476 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 34cef3aa11664e048d242d3feef47476; no voters:
I20260929 22:53:07.225625 14927 external_mini_cluster.cc:1428] Setting key 33120414b84914b8490607a19a2a4fae
I20260929 22:53:07.225720 30353 leader_election.cc:288] T 00000000000000000000000000000000 P 34cef3aa11664e048d242d3feef47476 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 22:53:07.225805 30356 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 34cef3aa11664e048d242d3feef47476 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 22:53:07.226001 30356 raft_consensus.cc:686] T 00000000000000000000000000000000 P 34cef3aa11664e048d242d3feef47476 [term 1 LEADER]: Becoming Leader. State: Replica: 34cef3aa11664e048d242d3feef47476, State: Running, Role: LEADER
I20260929 22:53:07.226023 30353 sys_catalog.cc:564] T 00000000000000000000000000000000 P 34cef3aa11664e048d242d3feef47476 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 22:53:07.226171 30356 consensus_queue.cc:237] T 00000000000000000000000000000000 P 34cef3aa11664e048d242d3feef47476 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "34cef3aa11664e048d242d3feef47476" member_type: VOTER last_known_addr { host: "127.14.147.254" port: 35531 } }
I20260929 22:53:07.226552 30356 sys_catalog.cc:455] T 00000000000000000000000000000000 P 34cef3aa11664e048d242d3feef47476 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 34cef3aa11664e048d242d3feef47476. Latest consensus state: current_term: 1 leader_uuid: "34cef3aa11664e048d242d3feef47476" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "34cef3aa11664e048d242d3feef47476" member_type: VOTER last_known_addr { host: "127.14.147.254" port: 35531 } } }
I20260929 22:53:07.226662 30356 sys_catalog.cc:458] T 00000000000000000000000000000000 P 34cef3aa11664e048d242d3feef47476 [sys.catalog]: This master's current role is: LEADER
I20260929 22:53:07.226562 30357 sys_catalog.cc:455] T 00000000000000000000000000000000 P 34cef3aa11664e048d242d3feef47476 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "34cef3aa11664e048d242d3feef47476" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "34cef3aa11664e048d242d3feef47476" member_type: VOTER last_known_addr { host: "127.14.147.254" port: 35531 } } }
I20260929 22:53:07.226760 30357 sys_catalog.cc:458] T 00000000000000000000000000000000 P 34cef3aa11664e048d242d3feef47476 [sys.catalog]: This master's current role is: LEADER
I20260929 22:53:07.227027 30366 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 22:53:07.227794 30366 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 22:53:07.229353 30366 catalog_manager.cc:1347] Generated new cluster ID: 8c16912f8a7f42e883c53e5491bfb100
I20260929 22:53:07.229405 30366 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 29 22:53:07 dist-test-slave-98mk krb5kdc[30272](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790722387, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.147.254@KRBTEST.COM
I20260929 22:53:07.234899 30366 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 22:53:07.235572 30366 catalog_manager.cc:1504] Loading token signing keys...
I20260929 22:53:07.238149 30362 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:53:07.226543 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53890 (local address 127.14.147.254:35531)
0929 22:53:07.226938 (+ 395us) server_negotiation.cc:206] Beginning negotiation
0929 22:53:07.226949 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0929 22:53:07.226987 (+ 38us) server_negotiation.cc:407] Connection header received
0929 22:53:07.227119 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 22:53:07.227127 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 22:53:07.227195 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 22:53:07.227339 (+ 144us) server_negotiation.cc:226] Negotiated authn=SASL
0929 22:53:07.228231 (+ 892us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:53:07.229611 (+ 1380us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:53:07.230714 (+ 1103us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:53:07.231382 (+ 668us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:53:07.231549 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 22:53:07.234294 (+ 2745us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 22:53:07.234300 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 22:53:07.234301 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 22:53:07.234318 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0929 22:53:07.236468 (+ 2150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 22:53:07.236876 (+ 408us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 22:53:07.236879 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 22:53:07.236880 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 22:53:07.237002 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 22:53:07.237182 (+ 180us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 22:53:07.237183 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 22:53:07.237183 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 22:53:07.237511 (+ 328us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 22:53:07.237668 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0929 22:53:07.237739 (+ 71us) server_negotiation.cc:299] Negotiation successful
0929 22:53:07.237919 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":340,"thread_start_us":57,"threads_started":1}
I20260929 22:53:07.241158 30366 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 34cef3aa11664e048d242d3feef47476: Generated new TSK 0
I20260929 22:53:07.241338 30366 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.14.147.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.147.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.147.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790722213368738-14927-0/krb5kdc/kudu_127.14.147.193.keytab.
Entry for principal kudu/127.14.147.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790722213368738-14927-0/krb5kdc/kudu_127.14.147.193.keytab.
I20260929 22:53:07.271818 14927 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhWOCLk/build/release/bin/kudu
/tmp/dist-test-taskhWOCLk/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790722213368738-14927-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790722213368738-14927-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790722213368738-14927-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790722213368738-14927-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.147.193:0
--local_ip_for_outbound_sockets=127.14.147.193
--webserver_interface=127.14.147.193
--webserver_port=0
--tserver_master_addrs=127.14.147.254:35531
--keytab_file=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790722213368738-14927-0/krb5kdc/kudu_127.14.147.193.keytab
--principal=kudu/127.14.147.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.147.212:41357
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790722213368738-14927-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790722213368738-14927-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790722213368738-14927-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 22:53:07.365643 30379 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 22:53:07.365955 30379 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 22:53:07.365999 30379 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 22:53:07.367635 30379 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 22:53:07.367683 30379 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 22:53:07.367726 30379 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.147.193
I20260929 22:53:07.370529 30379 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.147.212:41357
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790722213368738-14927-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790722213368738-14927-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790722213368738-14927-0/krb5kdc/kudu_127.14.147.193.keytab
--principal=kudu/127.14.147.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.147.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790722213368738-14927-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.14.147.193
--webserver_port=0
--tserver_master_addrs=127.14.147.254:35531
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30379
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.147.193
--log_dir=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790722213368738-14927-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 21:58:54 UTC on ccycloud.quasar-xguoti.root.comops.site
build id 4681390
I20260929 22:53:07.370899 30379 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 22:53:07.371147 30379 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 22:53:07.376263 30385 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 22:53:07.376312 30387 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 22:53:07.376405 30379 server_base.cc:1034] running on GCE node
W20260929 22:53:07.376266 30384 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 22:53:07.376766 30379 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 22:53:07.377039 30379 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 22:53:07.378221 30379 hybrid_clock.cc:648] HybridClock initialized: now 1790722387378201 us; error 28 us; skew 500 ppm
Sep 29 22:53:07 dist-test-slave-98mk krb5kdc[30272](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790722387, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.147.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 22:53:07.380782 30379 init.cc:377] Logged in from keytab as kudu/127.14.147.193@KRBTEST.COM (short username kudu)
I20260929 22:53:07.381214 30379 webserver.cc:460] Webserver started at http://127.14.147.193:44357/ using document root <none> and password file <none>
I20260929 22:53:07.381395 30379 fs_manager.cc:359] Metadata directory not provided
I20260929 22:53:07.381434 30379 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 22:53:07.381529 30379 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 22:53:07.382602 30379 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790722213368738-14927-0/minicluster-data/ts-0/data/instance:
uuid: "d1bd8776e4004c71ac8065dc47cf5725"
format_stamp: "Formatted at 2026-09-29 22:53:07 on dist-test-slave-98mk"
server_key: "73d4082730f9488ee6a939b600ad086f"
server_key_iv: "bfc24ed012d73abd308ad29d5d8007c3"
server_key_version: "encryptionkey@0"
I20260929 22:53:07.382910 30379 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790722213368738-14927-0/minicluster-data/ts-0/wal/instance:
uuid: "d1bd8776e4004c71ac8065dc47cf5725"
format_stamp: "Formatted at 2026-09-29 22:53:07 on dist-test-slave-98mk"
server_key: "73d4082730f9488ee6a939b600ad086f"
server_key_iv: "bfc24ed012d73abd308ad29d5d8007c3"
server_key_version: "encryptionkey@0"
I20260929 22:53:07.384277 30379 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260929 22:53:07.385143 30394 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 22:53:07.385326 30379 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260929 22:53:07.385416 30379 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790722213368738-14927-0/minicluster-data/ts-0/data,/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790722213368738-14927-0/minicluster-data/ts-0/wal
uuid: "d1bd8776e4004c71ac8065dc47cf5725"
format_stamp: "Formatted at 2026-09-29 22:53:07 on dist-test-slave-98mk"
server_key: "73d4082730f9488ee6a939b600ad086f"
server_key_iv: "bfc24ed012d73abd308ad29d5d8007c3"
server_key_version: "encryptionkey@0"
I20260929 22:53:07.385504 30379 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790722213368738-14927-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790722213368738-14927-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790722213368738-14927-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 22:53:07.408835 30379 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 22:53:07.411082 30379 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 22:53:07.411206 30379 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 22:53:07.411408 30379 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 22:53:07.411746 30379 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 22:53:07.411779 30379 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 22:53:07.411803 30379 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 22:53:07.411824 30379 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 22:53:07.417299 30379 rpc_server.cc:307] RPC server started. Bound to: 127.14.147.193:42211
I20260929 22:53:07.417372 30507 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.147.193:42211 every 8 connection(s)
I20260929 22:53:07.417668 30379 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790722213368738-14927-0/minicluster-data/ts-0/data/info.pb
Sep 29 22:53:07 dist-test-slave-98mk krb5kdc[30272](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790722387, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.147.193@KRBTEST.COM for kudu/127.14.147.254@KRBTEST.COM
I20260929 22:53:07.427887 14927 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhWOCLk/build/release/bin/kudu as pid 30379
I20260929 22:53:07.427989 14927 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790722213368738-14927-0/minicluster-data/ts-0/wal/instance
I20260929 22:53:07.428121 30362 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:53:07.418385 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.147.193:47605 (local address 127.14.147.254:35531)
0929 22:53:07.418431 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0929 22:53:07.418434 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 22:53:07.418675 (+ 241us) server_negotiation.cc:407] Connection header received
0929 22:53:07.419139 (+ 464us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 22:53:07.419140 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 22:53:07.419151 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 22:53:07.419215 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0929 22:53:07.420222 (+ 1007us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:53:07.420732 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:53:07.422102 (+ 1370us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:53:07.422320 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:53:07.422389 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 22:53:07.425240 (+ 2851us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 22:53:07.425242 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 22:53:07.425242 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 22:53:07.425250 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0929 22:53:07.426547 (+ 1297us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 22:53:07.427024 (+ 477us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 22:53:07.427025 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 22:53:07.427026 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 22:53:07.427085 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 22:53:07.427356 (+ 271us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 22:53:07.427356 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 22:53:07.427357 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 22:53:07.427513 (+ 156us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 22:53:07.427567 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0929 22:53:07.427894 (+ 327us) server_negotiation.cc:299] Negotiation successful
0929 22:53:07.428026 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260929 22:53:07.428166 14927 external_mini_cluster.cc:1428] Setting key 59fe220d1ad362a4cc83139c2a872245
I20260929 22:53:07.428274 30510 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:53:07.418342 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.147.254:35531 (local address 127.14.147.193:47605)
0929 22:53:07.418522 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0929 22:53:07.418566 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0929 22:53:07.419054 (+ 488us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 22:53:07.419297 (+ 243us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 22:53:07.419300 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 22:53:07.419917 (+ 617us) client_negotiation.cc:189] Negotiated authn=SASL
0929 22:53:07.420135 (+ 218us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 22:53:07.420140 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 22:53:07.420915 (+ 775us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 22:53:07.420917 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 22:53:07.421960 (+ 1043us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 22:53:07.421964 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 22:53:07.422477 (+ 513us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 22:53:07.422482 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 22:53:07.422651 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 22:53:07.423339 (+ 688us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 22:53:07.423350 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0929 22:53:07.425132 (+ 1782us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 22:53:07.426623 (+ 1491us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 22:53:07.426632 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 22:53:07.426634 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 22:53:07.426936 (+ 302us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 22:53:07.427173 (+ 237us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 22:53:07.427174 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 22:53:07.427174 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 22:53:07.427297 (+ 123us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 22:53:07.427572 (+ 275us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 22:53:07.427579 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 22:53:07.427750 (+ 171us) client_negotiation.cc:769] Sending connection context
0929 22:53:07.427851 (+ 101us) client_negotiation.cc:240] Negotiation successful
0929 22:53:07.428042 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":138,"thread_start_us":81,"threads_started":1}
I20260929 22:53:07.428758 30508 heartbeater.cc:347] Connected to a master server at 127.14.147.254:35531
I20260929 22:53:07.428866 30508 heartbeater.cc:464] Registering TS with master...
I20260929 22:53:07.429054 30508 heartbeater.cc:511] Master 127.14.147.254:35531 requested a full tablet report, sending...
I20260929 22:53:07.429558 30317 ts_manager.cc:194] Registered new tserver with Master: d1bd8776e4004c71ac8065dc47cf5725 (127.14.147.193:42211)
I20260929 22:53:07.430315 30317 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.147.193@KRBTEST.COM'} at 127.14.147.193:47605
WARNING: no policy specified for kudu/127.14.147.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.147.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.147.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790722213368738-14927-0/krb5kdc/kudu_127.14.147.194.keytab.
Entry for principal kudu/127.14.147.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790722213368738-14927-0/krb5kdc/kudu_127.14.147.194.keytab.
I20260929 22:53:07.449780 14927 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhWOCLk/build/release/bin/kudu
/tmp/dist-test-taskhWOCLk/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790722213368738-14927-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790722213368738-14927-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790722213368738-14927-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790722213368738-14927-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.147.194:0
--local_ip_for_outbound_sockets=127.14.147.194
--webserver_interface=127.14.147.194
--webserver_port=0
--tserver_master_addrs=127.14.147.254:35531
--keytab_file=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790722213368738-14927-0/krb5kdc/kudu_127.14.147.194.keytab
--principal=kudu/127.14.147.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.147.212:41357
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790722213368738-14927-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790722213368738-14927-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790722213368738-14927-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 22:53:07.532972 30513 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 22:53:07.533193 30513 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 22:53:07.533218 30513 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 22:53:07.534385 30513 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 22:53:07.534420 30513 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 22:53:07.534446 30513 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.147.194
I20260929 22:53:07.535895 30513 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.147.212:41357
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790722213368738-14927-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790722213368738-14927-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790722213368738-14927-0/krb5kdc/kudu_127.14.147.194.keytab
--principal=kudu/127.14.147.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.147.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790722213368738-14927-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.14.147.194
--webserver_port=0
--tserver_master_addrs=127.14.147.254:35531
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30513
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.147.194
--log_dir=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790722213368738-14927-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 21:58:54 UTC on ccycloud.quasar-xguoti.root.comops.site
build id 4681390
I20260929 22:53:07.536090 30513 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 22:53:07.536275 30513 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 22:53:07.541041 30521 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 22:53:07.541188 30513 server_base.cc:1034] running on GCE node
W20260929 22:53:07.541132 30518 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 22:53:07.541052 30519 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 22:53:07.541514 30513 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 22:53:07.541814 30513 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 22:53:07.542989 30513 hybrid_clock.cc:648] HybridClock initialized: now 1790722387542971 us; error 37 us; skew 500 ppm
Sep 29 22:53:07 dist-test-slave-98mk krb5kdc[30272](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790722387, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.147.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 22:53:07.545473 30513 init.cc:377] Logged in from keytab as kudu/127.14.147.194@KRBTEST.COM (short username kudu)
I20260929 22:53:07.545907 30513 webserver.cc:460] Webserver started at http://127.14.147.194:42367/ using document root <none> and password file <none>
I20260929 22:53:07.546103 30513 fs_manager.cc:359] Metadata directory not provided
I20260929 22:53:07.546149 30513 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 22:53:07.546271 30513 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 22:53:07.547299 30513 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790722213368738-14927-0/minicluster-data/ts-1/data/instance:
uuid: "28541c11b79044cf8708e09c266721fc"
format_stamp: "Formatted at 2026-09-29 22:53:07 on dist-test-slave-98mk"
server_key: "cfc2dc837f7660f34a1a910a5b5658bf"
server_key_iv: "f2f98582a6acbe96e50692ba98c1f05b"
server_key_version: "encryptionkey@0"
I20260929 22:53:07.547670 30513 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790722213368738-14927-0/minicluster-data/ts-1/wal/instance:
uuid: "28541c11b79044cf8708e09c266721fc"
format_stamp: "Formatted at 2026-09-29 22:53:07 on dist-test-slave-98mk"
server_key: "cfc2dc837f7660f34a1a910a5b5658bf"
server_key_iv: "f2f98582a6acbe96e50692ba98c1f05b"
server_key_version: "encryptionkey@0"
I20260929 22:53:07.548915 30513 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260929 22:53:07.549579 30528 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 22:53:07.549726 30513 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260929 22:53:07.549814 30513 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790722213368738-14927-0/minicluster-data/ts-1/data,/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790722213368738-14927-0/minicluster-data/ts-1/wal
uuid: "28541c11b79044cf8708e09c266721fc"
format_stamp: "Formatted at 2026-09-29 22:53:07 on dist-test-slave-98mk"
server_key: "cfc2dc837f7660f34a1a910a5b5658bf"
server_key_iv: "f2f98582a6acbe96e50692ba98c1f05b"
server_key_version: "encryptionkey@0"
I20260929 22:53:07.549885 30513 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790722213368738-14927-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790722213368738-14927-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790722213368738-14927-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 22:53:07.566709 30513 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 22:53:07.568941 30513 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 22:53:07.569087 30513 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 22:53:07.569319 30513 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 22:53:07.569643 30513 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 22:53:07.569675 30513 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 22:53:07.569705 30513 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 22:53:07.569728 30513 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 22:53:07.575403 30513 rpc_server.cc:307] RPC server started. Bound to: 127.14.147.194:46873
I20260929 22:53:07.575496 30641 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.147.194:46873 every 8 connection(s)
I20260929 22:53:07.575793 30513 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790722213368738-14927-0/minicluster-data/ts-1/data/info.pb
Sep 29 22:53:07 dist-test-slave-98mk krb5kdc[30272](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790722387, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.147.194@KRBTEST.COM for kudu/127.14.147.254@KRBTEST.COM
I20260929 22:53:07.584811 14927 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhWOCLk/build/release/bin/kudu as pid 30513
I20260929 22:53:07.584959 14927 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790722213368738-14927-0/minicluster-data/ts-1/wal/instance
I20260929 22:53:07.585222 14927 external_mini_cluster.cc:1428] Setting key e5e8f6a9555c4ad96030bb20717c7295
I20260929 22:53:07.585731 30362 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:53:07.576521 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.147.194:57471 (local address 127.14.147.254:35531)
0929 22:53:07.576569 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0929 22:53:07.576571 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 22:53:07.576747 (+ 176us) server_negotiation.cc:407] Connection header received
0929 22:53:07.577241 (+ 494us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 22:53:07.577242 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 22:53:07.577251 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 22:53:07.577309 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0929 22:53:07.578229 (+ 920us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:53:07.578744 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:53:07.579827 (+ 1083us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:53:07.580026 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:53:07.580089 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 22:53:07.582474 (+ 2385us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 22:53:07.582476 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 22:53:07.582476 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 22:53:07.582483 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0929 22:53:07.583965 (+ 1482us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 22:53:07.584531 (+ 566us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 22:53:07.584539 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 22:53:07.584540 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 22:53:07.584603 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 22:53:07.584939 (+ 336us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 22:53:07.584939 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 22:53:07.584939 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 22:53:07.585076 (+ 137us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 22:53:07.585134 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0929 22:53:07.585554 (+ 420us) server_negotiation.cc:299] Negotiation successful
0929 22:53:07.585640 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260929 22:53:07.586007 30644 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:53:07.576480 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.147.254:35531 (local address 127.14.147.194:57471)
0929 22:53:07.576627 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0929 22:53:07.576669 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0929 22:53:07.577160 (+ 491us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 22:53:07.577365 (+ 205us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 22:53:07.577368 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 22:53:07.577931 (+ 563us) client_negotiation.cc:189] Negotiated authn=SASL
0929 22:53:07.578128 (+ 197us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 22:53:07.578133 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 22:53:07.578847 (+ 714us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 22:53:07.578848 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 22:53:07.579728 (+ 880us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 22:53:07.579730 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 22:53:07.580095 (+ 365us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 22:53:07.580096 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 22:53:07.580190 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 22:53:07.580679 (+ 489us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 22:53:07.580688 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0929 22:53:07.582362 (+ 1674us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 22:53:07.584103 (+ 1741us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 22:53:07.584113 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 22:53:07.584116 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 22:53:07.584421 (+ 305us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 22:53:07.584677 (+ 256us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 22:53:07.584677 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 22:53:07.584678 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 22:53:07.584865 (+ 187us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 22:53:07.585146 (+ 281us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 22:53:07.585155 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 22:53:07.585381 (+ 226us) client_negotiation.cc:769] Sending connection context
0929 22:53:07.585503 (+ 122us) client_negotiation.cc:240] Negotiation successful
0929 22:53:07.585718 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":104,"thread_start_us":77,"threads_started":1}
I20260929 22:53:07.586632 30642 heartbeater.cc:347] Connected to a master server at 127.14.147.254:35531
I20260929 22:53:07.586748 30642 heartbeater.cc:464] Registering TS with master...
I20260929 22:53:07.586944 30642 heartbeater.cc:511] Master 127.14.147.254:35531 requested a full tablet report, sending...
I20260929 22:53:07.587322 30317 ts_manager.cc:194] Registered new tserver with Master: 28541c11b79044cf8708e09c266721fc (127.14.147.194:46873)
I20260929 22:53:07.587728 30317 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.147.194@KRBTEST.COM'} at 127.14.147.194:57471
WARNING: no policy specified for kudu/127.14.147.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.147.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.147.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790722213368738-14927-0/krb5kdc/kudu_127.14.147.195.keytab.
Entry for principal kudu/127.14.147.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790722213368738-14927-0/krb5kdc/kudu_127.14.147.195.keytab.
I20260929 22:53:07.608650 14927 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhWOCLk/build/release/bin/kudu
/tmp/dist-test-taskhWOCLk/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790722213368738-14927-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790722213368738-14927-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790722213368738-14927-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790722213368738-14927-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.147.195:0
--local_ip_for_outbound_sockets=127.14.147.195
--webserver_interface=127.14.147.195
--webserver_port=0
--tserver_master_addrs=127.14.147.254:35531
--keytab_file=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790722213368738-14927-0/krb5kdc/kudu_127.14.147.195.keytab
--principal=kudu/127.14.147.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.147.212:41357
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790722213368738-14927-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790722213368738-14927-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790722213368738-14927-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 22:53:07.688921 30647 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 22:53:07.689139 30647 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 22:53:07.689172 30647 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 22:53:07.690377 30647 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 22:53:07.690420 30647 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 22:53:07.690457 30647 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.147.195
I20260929 22:53:07.692050 30647 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.147.212:41357
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790722213368738-14927-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790722213368738-14927-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790722213368738-14927-0/krb5kdc/kudu_127.14.147.195.keytab
--principal=kudu/127.14.147.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.147.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790722213368738-14927-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.14.147.195
--webserver_port=0
--tserver_master_addrs=127.14.147.254:35531
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30647
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.147.195
--log_dir=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790722213368738-14927-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 21:58:54 UTC on ccycloud.quasar-xguoti.root.comops.site
build id 4681390
I20260929 22:53:07.692299 30647 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 22:53:07.692513 30647 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 22:53:07.697373 30652 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 22:53:07.697363 30655 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 22:53:07.697446 30647 server_base.cc:1034] running on GCE node
W20260929 22:53:07.697364 30653 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 22:53:07.697780 30647 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 22:53:07.698062 30647 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 22:53:07.699225 30647 hybrid_clock.cc:648] HybridClock initialized: now 1790722387699205 us; error 36 us; skew 500 ppm
Sep 29 22:53:07 dist-test-slave-98mk krb5kdc[30272](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790722387, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.147.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 22:53:07.701718 30647 init.cc:377] Logged in from keytab as kudu/127.14.147.195@KRBTEST.COM (short username kudu)
I20260929 22:53:07.702150 30647 webserver.cc:460] Webserver started at http://127.14.147.195:40869/ using document root <none> and password file <none>
I20260929 22:53:07.702365 30647 fs_manager.cc:359] Metadata directory not provided
I20260929 22:53:07.702411 30647 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 22:53:07.702517 30647 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 22:53:07.703619 30647 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790722213368738-14927-0/minicluster-data/ts-2/data/instance:
uuid: "4abdea75838e4fbd8bf5d3f863eafea7"
format_stamp: "Formatted at 2026-09-29 22:53:07 on dist-test-slave-98mk"
server_key: "1da1349444d8dccc82d5d454eec06bef"
server_key_iv: "5cdc9391c1f1db9bf81ed2981898a8d6"
server_key_version: "encryptionkey@0"
I20260929 22:53:07.703953 30647 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790722213368738-14927-0/minicluster-data/ts-2/wal/instance:
uuid: "4abdea75838e4fbd8bf5d3f863eafea7"
format_stamp: "Formatted at 2026-09-29 22:53:07 on dist-test-slave-98mk"
server_key: "1da1349444d8dccc82d5d454eec06bef"
server_key_iv: "5cdc9391c1f1db9bf81ed2981898a8d6"
server_key_version: "encryptionkey@0"
I20260929 22:53:07.705268 30647 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260929 22:53:07.706068 30662 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 22:53:07.706226 30647 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260929 22:53:07.706302 30647 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790722213368738-14927-0/minicluster-data/ts-2/data,/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790722213368738-14927-0/minicluster-data/ts-2/wal
uuid: "4abdea75838e4fbd8bf5d3f863eafea7"
format_stamp: "Formatted at 2026-09-29 22:53:07 on dist-test-slave-98mk"
server_key: "1da1349444d8dccc82d5d454eec06bef"
server_key_iv: "5cdc9391c1f1db9bf81ed2981898a8d6"
server_key_version: "encryptionkey@0"
I20260929 22:53:07.706380 30647 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790722213368738-14927-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790722213368738-14927-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790722213368738-14927-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 22:53:07.720412 30647 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 22:53:07.722355 30647 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 22:53:07.722496 30647 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 22:53:07.722724 30647 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 22:53:07.723026 30647 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 22:53:07.723057 30647 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 22:53:07.723088 30647 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 22:53:07.723111 30647 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 22:53:07.729050 30647 rpc_server.cc:307] RPC server started. Bound to: 127.14.147.195:40667
I20260929 22:53:07.729127 30775 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.147.195:40667 every 8 connection(s)
I20260929 22:53:07.729405 30647 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790722213368738-14927-0/minicluster-data/ts-2/data/info.pb
I20260929 22:53:07.733768 14927 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhWOCLk/build/release/bin/kudu as pid 30647
I20260929 22:53:07.733883 14927 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790722213368738-14927-0/minicluster-data/ts-2/wal/instance
I20260929 22:53:07.734112 14927 external_mini_cluster.cc:1428] Setting key 378b1ebe6ef2f6e6a8fffe7ec4ea41c5
Sep 29 22:53:07 dist-test-slave-98mk krb5kdc[30272](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790722387, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.147.195@KRBTEST.COM for kudu/127.14.147.254@KRBTEST.COM
I20260929 22:53:07.738343 30362 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:53:07.730125 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.147.195:38581 (local address 127.14.147.254:35531)
0929 22:53:07.730199 (+ 74us) server_negotiation.cc:206] Beginning negotiation
0929 22:53:07.730201 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 22:53:07.730326 (+ 125us) server_negotiation.cc:407] Connection header received
0929 22:53:07.730836 (+ 510us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 22:53:07.730837 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 22:53:07.730848 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 22:53:07.730904 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0929 22:53:07.731869 (+ 965us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:53:07.732377 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:53:07.733338 (+ 961us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:53:07.733570 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:53:07.733634 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 22:53:07.735997 (+ 2363us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 22:53:07.735998 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 22:53:07.735999 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 22:53:07.736006 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0929 22:53:07.737083 (+ 1077us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 22:53:07.737417 (+ 334us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 22:53:07.737417 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 22:53:07.737418 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 22:53:07.737464 (+ 46us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 22:53:07.737693 (+ 229us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 22:53:07.737694 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 22:53:07.737694 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 22:53:07.737821 (+ 127us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 22:53:07.737881 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0929 22:53:07.738165 (+ 284us) server_negotiation.cc:299] Negotiation successful
0929 22:53:07.738252 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260929 22:53:07.738525 30778 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:53:07.730054 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.147.254:35531 (local address 127.14.147.195:38581)
0929 22:53:07.730212 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0929 22:53:07.730250 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0929 22:53:07.730725 (+ 475us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 22:53:07.730956 (+ 231us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 22:53:07.730959 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 22:53:07.731587 (+ 628us) client_negotiation.cc:189] Negotiated authn=SASL
0929 22:53:07.731786 (+ 199us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 22:53:07.731791 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 22:53:07.732460 (+ 669us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 22:53:07.732460 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 22:53:07.733252 (+ 792us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 22:53:07.733254 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 22:53:07.733673 (+ 419us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 22:53:07.733675 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 22:53:07.733850 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 22:53:07.734426 (+ 576us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 22:53:07.734436 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0929 22:53:07.735918 (+ 1482us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 22:53:07.737146 (+ 1228us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 22:53:07.737152 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 22:53:07.737154 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 22:53:07.737360 (+ 206us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 22:53:07.737510 (+ 150us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 22:53:07.737510 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 22:53:07.737511 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 22:53:07.737626 (+ 115us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 22:53:07.737880 (+ 254us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 22:53:07.737887 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 22:53:07.738042 (+ 155us) client_negotiation.cc:769] Sending connection context
0929 22:53:07.738130 (+ 88us) client_negotiation.cc:240] Negotiation successful
0929 22:53:07.738291 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":65,"threads_started":1}
I20260929 22:53:07.739004 30776 heartbeater.cc:347] Connected to a master server at 127.14.147.254:35531
I20260929 22:53:07.739091 30776 heartbeater.cc:464] Registering TS with master...
I20260929 22:53:07.739264 30776 heartbeater.cc:511] Master 127.14.147.254:35531 requested a full tablet report, sending...
I20260929 22:53:07.739674 30317 ts_manager.cc:194] Registered new tserver with Master: 4abdea75838e4fbd8bf5d3f863eafea7 (127.14.147.195:40667)
I20260929 22:53:07.740028 30317 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.147.195@KRBTEST.COM'} at 127.14.147.195:38581
I20260929 22:53:07.745843 14927 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 22:53:07.753983 30362 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:53:07.747101 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53894 (local address 127.14.147.254:35531)
0929 22:53:07.747254 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0929 22:53:07.747257 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 22:53:07.747408 (+ 151us) server_negotiation.cc:407] Connection header received
0929 22:53:07.747544 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 22:53:07.747545 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 22:53:07.747558 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 22:53:07.747613 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0929 22:53:07.748282 (+ 669us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:53:07.748847 (+ 565us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:53:07.749578 (+ 731us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:53:07.749819 (+ 241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:53:07.749891 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 22:53:07.751369 (+ 1478us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 22:53:07.751371 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 22:53:07.751372 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 22:53:07.751382 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0929 22:53:07.752820 (+ 1438us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 22:53:07.753163 (+ 343us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 22:53:07.753164 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 22:53:07.753165 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 22:53:07.753221 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 22:53:07.753379 (+ 158us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 22:53:07.753380 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 22:53:07.753380 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 22:53:07.753560 (+ 180us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 22:53:07.753647 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0929 22:53:07.753796 (+ 149us) server_negotiation.cc:299] Negotiation successful
0929 22:53:07.753885 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":97}
I20260929 22:53:07.755677 30317 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53894:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 22:53:07.757265 30317 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 22:53:07.764163 30789 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:53:07.759979 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59636 (local address 127.14.147.195:40667)
0929 22:53:07.760167 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0929 22:53:07.760173 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 22:53:07.760325 (+ 152us) server_negotiation.cc:407] Connection header received
0929 22:53:07.760445 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 22:53:07.760448 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 22:53:07.760465 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 22:53:07.760546 (+ 81us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 22:53:07.760733 (+ 187us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:53:07.761908 (+ 1175us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:53:07.763164 (+ 1256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:53:07.763740 (+ 576us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:53:07.763811 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 22:53:07.763873 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0929 22:53:07.764022 (+ 149us) server_negotiation.cc:299] Negotiation successful
0929 22:53:07.764068 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":151,"thread_start_us":100,"threads_started":1}
I20260929 22:53:07.764153 30791 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:53:07.759902 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.147.195:40667 (local address 127.0.0.1:59636)
0929 22:53:07.760221 (+ 319us) negotiation.cc:107] Waiting for socket to connect
0929 22:53:07.760254 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0929 22:53:07.760356 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 22:53:07.760559 (+ 203us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 22:53:07.760561 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 22:53:07.760563 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 22:53:07.760648 (+ 85us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 22:53:07.760651 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 22:53:07.762020 (+ 1369us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 22:53:07.762020 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 22:53:07.763091 (+ 1071us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 22:53:07.763093 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 22:53:07.763816 (+ 723us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 22:53:07.763817 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 22:53:07.763914 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 22:53:07.763930 (+ 16us) client_negotiation.cc:769] Sending connection context
0929 22:53:07.764004 (+ 74us) client_negotiation.cc:240] Negotiation successful
0929 22:53:07.764067 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":296,"mutex_wait_us":28,"thread_start_us":91,"threads_started":1}
I20260929 22:53:07.764330 30792 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:53:07.760335 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37700 (local address 127.14.147.194:46873)
0929 22:53:07.760501 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0929 22:53:07.760505 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 22:53:07.760513 (+ 8us) server_negotiation.cc:407] Connection header received
0929 22:53:07.760550 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 22:53:07.760553 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 22:53:07.760567 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 22:53:07.760631 (+ 64us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 22:53:07.760841 (+ 210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:53:07.761561 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:53:07.763008 (+ 1447us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:53:07.763803 (+ 795us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:53:07.763874 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 22:53:07.763927 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0929 22:53:07.764196 (+ 269us) server_negotiation.cc:299] Negotiation successful
0929 22:53:07.764242 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":124,"thread_start_us":86,"threads_started":1}
I20260929 22:53:07.764330 30790 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:53:07.760025 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53558 (local address 127.14.147.193:42211)
0929 22:53:07.760254 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0929 22:53:07.760257 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 22:53:07.760487 (+ 230us) server_negotiation.cc:407] Connection header received
0929 22:53:07.760546 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 22:53:07.760549 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 22:53:07.760565 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 22:53:07.760705 (+ 140us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 22:53:07.760935 (+ 230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:53:07.761993 (+ 1058us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:53:07.763016 (+ 1023us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:53:07.763803 (+ 787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:53:07.763874 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 22:53:07.763927 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0929 22:53:07.764195 (+ 268us) server_negotiation.cc:299] Negotiation successful
0929 22:53:07.764238 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":195,"thread_start_us":72,"threads_started":1}
I20260929 22:53:07.764395 30787 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:53:07.759902 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.147.193:42211 (local address 127.0.0.1:53558)
0929 22:53:07.760209 (+ 307us) negotiation.cc:107] Waiting for socket to connect
0929 22:53:07.760254 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0929 22:53:07.760332 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 22:53:07.760784 (+ 452us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 22:53:07.760784 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 22:53:07.760787 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 22:53:07.760823 (+ 36us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 22:53:07.760824 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 22:53:07.762069 (+ 1245us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 22:53:07.762070 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 22:53:07.762931 (+ 861us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 22:53:07.762934 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 22:53:07.763965 (+ 1031us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 22:53:07.763966 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 22:53:07.764056 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 22:53:07.764059 (+ 3us) client_negotiation.cc:769] Sending connection context
0929 22:53:07.764103 (+ 44us) client_negotiation.cc:240] Negotiation successful
0929 22:53:07.764147 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":280,"mutex_wait_us":1,"thread_start_us":68,"threads_started":1}
I20260929 22:53:07.764488 30788 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:53:07.760113 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.147.194:46873 (local address 127.0.0.1:37700)
0929 22:53:07.760433 (+ 320us) negotiation.cc:107] Waiting for socket to connect
0929 22:53:07.760442 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0929 22:53:07.760476 (+ 34us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 22:53:07.760707 (+ 231us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 22:53:07.760708 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 22:53:07.760711 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 22:53:07.760766 (+ 55us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 22:53:07.760768 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 22:53:07.761666 (+ 898us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 22:53:07.761667 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 22:53:07.762915 (+ 1248us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 22:53:07.762918 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 22:53:07.763963 (+ 1045us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 22:53:07.763965 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 22:53:07.764050 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 22:53:07.764055 (+ 5us) client_negotiation.cc:769] Sending connection context
0929 22:53:07.764103 (+ 48us) client_negotiation.cc:240] Negotiation successful
0929 22:53:07.764146 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":304,"mutex_wait_us":20,"thread_start_us":67,"threads_started":1}
I20260929 22:53:07.765265 30576 tablet_service.cc:1467] Processing CreateTablet for tablet c25404046ba247f491276b52af370a4f (DEFAULT_TABLE table=test-table [id=3335ce09a5de45cf9b202aef64333e9c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 22:53:07.765352 30710 tablet_service.cc:1467] Processing CreateTablet for tablet c25404046ba247f491276b52af370a4f (DEFAULT_TABLE table=test-table [id=3335ce09a5de45cf9b202aef64333e9c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 22:53:07.765622 30576 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c25404046ba247f491276b52af370a4f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 22:53:07.765651 30710 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c25404046ba247f491276b52af370a4f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 22:53:07.767910 30442 tablet_service.cc:1467] Processing CreateTablet for tablet c25404046ba247f491276b52af370a4f (DEFAULT_TABLE table=test-table [id=3335ce09a5de45cf9b202aef64333e9c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 22:53:07.768270 30442 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c25404046ba247f491276b52af370a4f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 22:53:07.768363 30793 tablet_bootstrap.cc:492] T c25404046ba247f491276b52af370a4f P 28541c11b79044cf8708e09c266721fc: Bootstrap starting.
I20260929 22:53:07.768937 30793 tablet_bootstrap.cc:654] T c25404046ba247f491276b52af370a4f P 28541c11b79044cf8708e09c266721fc: Neither blocks nor log segments found. Creating new log.
I20260929 22:53:07.769326 30793 log.cc:824] T c25404046ba247f491276b52af370a4f P 28541c11b79044cf8708e09c266721fc: Log is configured to *not* fsync() on all Append() calls
I20260929 22:53:07.769637 30794 tablet_bootstrap.cc:492] T c25404046ba247f491276b52af370a4f P 4abdea75838e4fbd8bf5d3f863eafea7: Bootstrap starting.
I20260929 22:53:07.770108 30793 tablet_bootstrap.cc:492] T c25404046ba247f491276b52af370a4f P 28541c11b79044cf8708e09c266721fc: No bootstrap required, opened a new log
I20260929 22:53:07.770162 30794 tablet_bootstrap.cc:654] T c25404046ba247f491276b52af370a4f P 4abdea75838e4fbd8bf5d3f863eafea7: Neither blocks nor log segments found. Creating new log.
I20260929 22:53:07.770169 30793 ts_tablet_manager.cc:1397] T c25404046ba247f491276b52af370a4f P 28541c11b79044cf8708e09c266721fc: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260929 22:53:07.770550 30794 log.cc:824] T c25404046ba247f491276b52af370a4f P 4abdea75838e4fbd8bf5d3f863eafea7: Log is configured to *not* fsync() on all Append() calls
I20260929 22:53:07.771142 30797 tablet_bootstrap.cc:492] T c25404046ba247f491276b52af370a4f P d1bd8776e4004c71ac8065dc47cf5725: Bootstrap starting.
I20260929 22:53:07.771670 30794 tablet_bootstrap.cc:492] T c25404046ba247f491276b52af370a4f P 4abdea75838e4fbd8bf5d3f863eafea7: No bootstrap required, opened a new log
I20260929 22:53:07.771745 30794 ts_tablet_manager.cc:1397] T c25404046ba247f491276b52af370a4f P 4abdea75838e4fbd8bf5d3f863eafea7: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260929 22:53:07.771973 30797 tablet_bootstrap.cc:654] T c25404046ba247f491276b52af370a4f P d1bd8776e4004c71ac8065dc47cf5725: Neither blocks nor log segments found. Creating new log.
I20260929 22:53:07.772141 30793 raft_consensus.cc:348] T c25404046ba247f491276b52af370a4f P 28541c11b79044cf8708e09c266721fc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1bd8776e4004c71ac8065dc47cf5725" member_type: VOTER last_known_addr { host: "127.14.147.193" port: 42211 } } peers { permanent_uuid: "28541c11b79044cf8708e09c266721fc" member_type: VOTER last_known_addr { host: "127.14.147.194" port: 46873 } } peers { permanent_uuid: "4abdea75838e4fbd8bf5d3f863eafea7" member_type: VOTER last_known_addr { host: "127.14.147.195" port: 40667 } }
I20260929 22:53:07.772294 30793 raft_consensus.cc:374] T c25404046ba247f491276b52af370a4f P 28541c11b79044cf8708e09c266721fc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 22:53:07.772327 30793 raft_consensus.cc:729] T c25404046ba247f491276b52af370a4f P 28541c11b79044cf8708e09c266721fc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 28541c11b79044cf8708e09c266721fc, State: Initialized, Role: FOLLOWER
I20260929 22:53:07.772507 30793 consensus_queue.cc:260] T c25404046ba247f491276b52af370a4f P 28541c11b79044cf8708e09c266721fc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1bd8776e4004c71ac8065dc47cf5725" member_type: VOTER last_known_addr { host: "127.14.147.193" port: 42211 } } peers { permanent_uuid: "28541c11b79044cf8708e09c266721fc" member_type: VOTER last_known_addr { host: "127.14.147.194" port: 46873 } } peers { permanent_uuid: "4abdea75838e4fbd8bf5d3f863eafea7" member_type: VOTER last_known_addr { host: "127.14.147.195" port: 40667 } }
I20260929 22:53:07.772527 30797 log.cc:824] T c25404046ba247f491276b52af370a4f P d1bd8776e4004c71ac8065dc47cf5725: Log is configured to *not* fsync() on all Append() calls
I20260929 22:53:07.772766 30793 ts_tablet_manager.cc:1428] T c25404046ba247f491276b52af370a4f P 28541c11b79044cf8708e09c266721fc: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260929 22:53:07.772874 30642 heartbeater.cc:503] Master 127.14.147.254:35531 was elected leader, sending a full tablet report...
I20260929 22:53:07.773435 30797 tablet_bootstrap.cc:492] T c25404046ba247f491276b52af370a4f P d1bd8776e4004c71ac8065dc47cf5725: No bootstrap required, opened a new log
I20260929 22:53:07.773510 30797 ts_tablet_manager.cc:1397] T c25404046ba247f491276b52af370a4f P d1bd8776e4004c71ac8065dc47cf5725: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260929 22:53:07.773783 30794 raft_consensus.cc:348] T c25404046ba247f491276b52af370a4f P 4abdea75838e4fbd8bf5d3f863eafea7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1bd8776e4004c71ac8065dc47cf5725" member_type: VOTER last_known_addr { host: "127.14.147.193" port: 42211 } } peers { permanent_uuid: "28541c11b79044cf8708e09c266721fc" member_type: VOTER last_known_addr { host: "127.14.147.194" port: 46873 } } peers { permanent_uuid: "4abdea75838e4fbd8bf5d3f863eafea7" member_type: VOTER last_known_addr { host: "127.14.147.195" port: 40667 } }
I20260929 22:53:07.773945 30794 raft_consensus.cc:374] T c25404046ba247f491276b52af370a4f P 4abdea75838e4fbd8bf5d3f863eafea7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 22:53:07.773975 30794 raft_consensus.cc:729] T c25404046ba247f491276b52af370a4f P 4abdea75838e4fbd8bf5d3f863eafea7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4abdea75838e4fbd8bf5d3f863eafea7, State: Initialized, Role: FOLLOWER
I20260929 22:53:07.774084 30794 consensus_queue.cc:260] T c25404046ba247f491276b52af370a4f P 4abdea75838e4fbd8bf5d3f863eafea7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1bd8776e4004c71ac8065dc47cf5725" member_type: VOTER last_known_addr { host: "127.14.147.193" port: 42211 } } peers { permanent_uuid: "28541c11b79044cf8708e09c266721fc" member_type: VOTER last_known_addr { host: "127.14.147.194" port: 46873 } } peers { permanent_uuid: "4abdea75838e4fbd8bf5d3f863eafea7" member_type: VOTER last_known_addr { host: "127.14.147.195" port: 40667 } }
I20260929 22:53:07.774353 30794 ts_tablet_manager.cc:1428] T c25404046ba247f491276b52af370a4f P 4abdea75838e4fbd8bf5d3f863eafea7: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260929 22:53:07.774379 30776 heartbeater.cc:503] Master 127.14.147.254:35531 was elected leader, sending a full tablet report...
I20260929 22:53:07.775467 30797 raft_consensus.cc:348] T c25404046ba247f491276b52af370a4f P d1bd8776e4004c71ac8065dc47cf5725 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1bd8776e4004c71ac8065dc47cf5725" member_type: VOTER last_known_addr { host: "127.14.147.193" port: 42211 } } peers { permanent_uuid: "28541c11b79044cf8708e09c266721fc" member_type: VOTER last_known_addr { host: "127.14.147.194" port: 46873 } } peers { permanent_uuid: "4abdea75838e4fbd8bf5d3f863eafea7" member_type: VOTER last_known_addr { host: "127.14.147.195" port: 40667 } }
I20260929 22:53:07.775651 30797 raft_consensus.cc:374] T c25404046ba247f491276b52af370a4f P d1bd8776e4004c71ac8065dc47cf5725 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 22:53:07.775688 30797 raft_consensus.cc:729] T c25404046ba247f491276b52af370a4f P d1bd8776e4004c71ac8065dc47cf5725 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d1bd8776e4004c71ac8065dc47cf5725, State: Initialized, Role: FOLLOWER
I20260929 22:53:07.775794 30797 consensus_queue.cc:260] T c25404046ba247f491276b52af370a4f P d1bd8776e4004c71ac8065dc47cf5725 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1bd8776e4004c71ac8065dc47cf5725" member_type: VOTER last_known_addr { host: "127.14.147.193" port: 42211 } } peers { permanent_uuid: "28541c11b79044cf8708e09c266721fc" member_type: VOTER last_known_addr { host: "127.14.147.194" port: 46873 } } peers { permanent_uuid: "4abdea75838e4fbd8bf5d3f863eafea7" member_type: VOTER last_known_addr { host: "127.14.147.195" port: 40667 } }
I20260929 22:53:07.776072 30797 ts_tablet_manager.cc:1428] T c25404046ba247f491276b52af370a4f P d1bd8776e4004c71ac8065dc47cf5725: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260929 22:53:07.776085 30508 heartbeater.cc:503] Master 127.14.147.254:35531 was elected leader, sending a full tablet report...
I20260929 22:53:07.800438 30800 raft_consensus.cc:482] T c25404046ba247f491276b52af370a4f P 4abdea75838e4fbd8bf5d3f863eafea7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 22:53:07.800592 30800 raft_consensus.cc:504] T c25404046ba247f491276b52af370a4f P 4abdea75838e4fbd8bf5d3f863eafea7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1bd8776e4004c71ac8065dc47cf5725" member_type: VOTER last_known_addr { host: "127.14.147.193" port: 42211 } } peers { permanent_uuid: "28541c11b79044cf8708e09c266721fc" member_type: VOTER last_known_addr { host: "127.14.147.194" port: 46873 } } peers { permanent_uuid: "4abdea75838e4fbd8bf5d3f863eafea7" member_type: VOTER last_known_addr { host: "127.14.147.195" port: 40667 } }
I20260929 22:53:07.800946 30800 leader_election.cc:288] T c25404046ba247f491276b52af370a4f P 4abdea75838e4fbd8bf5d3f863eafea7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d1bd8776e4004c71ac8065dc47cf5725 (127.14.147.193:42211), 28541c11b79044cf8708e09c266721fc (127.14.147.194:46873)
I20260929 22:53:07.804108 30790 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:53:07.801216 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.147.195:51989 (local address 127.14.147.193:42211)
0929 22:53:07.801260 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0929 22:53:07.801262 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 22:53:07.801286 (+ 24us) server_negotiation.cc:407] Connection header received
0929 22:53:07.801316 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 22:53:07.801317 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 22:53:07.801325 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 22:53:07.801370 (+ 45us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 22:53:07.801562 (+ 192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:53:07.802064 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:53:07.803135 (+ 1071us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:53:07.803584 (+ 449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:53:07.803644 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 22:53:07.803652 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0929 22:53:07.803976 (+ 324us) server_negotiation.cc:299] Negotiation successful
0929 22:53:07.804014 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260929 22:53:07.804102 30778 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:53:07.801131 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.147.193:42211 (local address 127.14.147.195:51989)
0929 22:53:07.801186 (+ 55us) negotiation.cc:107] Waiting for socket to connect
0929 22:53:07.801200 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 22:53:07.801262 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 22:53:07.801425 (+ 163us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 22:53:07.801426 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 22:53:07.801429 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 22:53:07.801505 (+ 76us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 22:53:07.801507 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 22:53:07.802145 (+ 638us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 22:53:07.802146 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 22:53:07.803046 (+ 900us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 22:53:07.803048 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 22:53:07.803712 (+ 664us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 22:53:07.803715 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 22:53:07.803839 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 22:53:07.803846 (+ 7us) client_negotiation.cc:769] Sending connection context
0929 22:53:07.803951 (+ 105us) client_negotiation.cc:240] Negotiation successful
0929 22:53:07.804014 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":31}
I20260929 22:53:07.804615 30462 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c25404046ba247f491276b52af370a4f" candidate_uuid: "4abdea75838e4fbd8bf5d3f863eafea7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1bd8776e4004c71ac8065dc47cf5725" is_pre_election: true
I20260929 22:53:07.804864 30462 raft_consensus.cc:2454] T c25404046ba247f491276b52af370a4f P d1bd8776e4004c71ac8065dc47cf5725 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4abdea75838e4fbd8bf5d3f863eafea7 in term 0.
I20260929 22:53:07.804930 30802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:53:07.801232 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.147.194:46873 (local address 127.14.147.195:58667)
0929 22:53:07.801405 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0929 22:53:07.801415 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0929 22:53:07.801463 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 22:53:07.802166 (+ 703us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 22:53:07.802167 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 22:53:07.802169 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 22:53:07.802234 (+ 65us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 22:53:07.802235 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 22:53:07.802827 (+ 592us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 22:53:07.802828 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 22:53:07.803897 (+ 1069us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 22:53:07.803899 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 22:53:07.804637 (+ 738us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 22:53:07.804638 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 22:53:07.804734 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 22:53:07.804739 (+ 5us) client_negotiation.cc:769] Sending connection context
0929 22:53:07.804802 (+ 63us) client_negotiation.cc:240] Negotiation successful
0929 22:53:07.804848 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":155,"thread_start_us":115,"threads_started":1}
I20260929 22:53:07.804939 30792 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:53:07.801419 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.147.195:58667 (local address 127.14.147.194:46873)
0929 22:53:07.801474 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0929 22:53:07.801475 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0929 22:53:07.801483 (+ 8us) server_negotiation.cc:407] Connection header received
0929 22:53:07.801506 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 22:53:07.801506 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 22:53:07.801512 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 22:53:07.801554 (+ 42us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 22:53:07.802297 (+ 743us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:53:07.802748 (+ 451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:53:07.804003 (+ 1255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:53:07.804542 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:53:07.804609 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 22:53:07.804616 (+ 7us) server_negotiation.cc:1035] Waiting for connection context
0929 22:53:07.804829 (+ 213us) server_negotiation.cc:299] Negotiation successful
0929 22:53:07.804876 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260929 22:53:07.805279 30666 leader_election.cc:302] T c25404046ba247f491276b52af370a4f P 4abdea75838e4fbd8bf5d3f863eafea7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4abdea75838e4fbd8bf5d3f863eafea7, d1bd8776e4004c71ac8065dc47cf5725; no voters:
I20260929 22:53:07.805433 30800 raft_consensus.cc:2781] T c25404046ba247f491276b52af370a4f P 4abdea75838e4fbd8bf5d3f863eafea7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 22:53:07.805505 30800 raft_consensus.cc:482] T c25404046ba247f491276b52af370a4f P 4abdea75838e4fbd8bf5d3f863eafea7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 22:53:07.805531 30800 raft_consensus.cc:3037] T c25404046ba247f491276b52af370a4f P 4abdea75838e4fbd8bf5d3f863eafea7 [term 0 FOLLOWER]: Advancing to term 1
I20260929 22:53:07.805737 30596 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c25404046ba247f491276b52af370a4f" candidate_uuid: "4abdea75838e4fbd8bf5d3f863eafea7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "28541c11b79044cf8708e09c266721fc" is_pre_election: true
I20260929 22:53:07.805910 30596 raft_consensus.cc:2454] T c25404046ba247f491276b52af370a4f P 28541c11b79044cf8708e09c266721fc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4abdea75838e4fbd8bf5d3f863eafea7 in term 0.
I20260929 22:53:07.806435 30800 raft_consensus.cc:504] T c25404046ba247f491276b52af370a4f P 4abdea75838e4fbd8bf5d3f863eafea7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1bd8776e4004c71ac8065dc47cf5725" member_type: VOTER last_known_addr { host: "127.14.147.193" port: 42211 } } peers { permanent_uuid: "28541c11b79044cf8708e09c266721fc" member_type: VOTER last_known_addr { host: "127.14.147.194" port: 46873 } } peers { permanent_uuid: "4abdea75838e4fbd8bf5d3f863eafea7" member_type: VOTER last_known_addr { host: "127.14.147.195" port: 40667 } }
I20260929 22:53:07.806579 30800 leader_election.cc:288] T c25404046ba247f491276b52af370a4f P 4abdea75838e4fbd8bf5d3f863eafea7 [CANDIDATE]: Term 1 election: Requested vote from peers d1bd8776e4004c71ac8065dc47cf5725 (127.14.147.193:42211), 28541c11b79044cf8708e09c266721fc (127.14.147.194:46873)
I20260929 22:53:07.806805 30462 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c25404046ba247f491276b52af370a4f" candidate_uuid: "4abdea75838e4fbd8bf5d3f863eafea7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1bd8776e4004c71ac8065dc47cf5725"
I20260929 22:53:07.806843 30596 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c25404046ba247f491276b52af370a4f" candidate_uuid: "4abdea75838e4fbd8bf5d3f863eafea7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "28541c11b79044cf8708e09c266721fc"
I20260929 22:53:07.806924 30596 raft_consensus.cc:3037] T c25404046ba247f491276b52af370a4f P 28541c11b79044cf8708e09c266721fc [term 0 FOLLOWER]: Advancing to term 1
I20260929 22:53:07.806924 30462 raft_consensus.cc:3037] T c25404046ba247f491276b52af370a4f P d1bd8776e4004c71ac8065dc47cf5725 [term 0 FOLLOWER]: Advancing to term 1
I20260929 22:53:07.808158 30462 raft_consensus.cc:2454] T c25404046ba247f491276b52af370a4f P d1bd8776e4004c71ac8065dc47cf5725 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4abdea75838e4fbd8bf5d3f863eafea7 in term 1.
I20260929 22:53:07.808158 30596 raft_consensus.cc:2454] T c25404046ba247f491276b52af370a4f P 28541c11b79044cf8708e09c266721fc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4abdea75838e4fbd8bf5d3f863eafea7 in term 1.
I20260929 22:53:07.808420 30666 leader_election.cc:302] T c25404046ba247f491276b52af370a4f P 4abdea75838e4fbd8bf5d3f863eafea7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4abdea75838e4fbd8bf5d3f863eafea7, d1bd8776e4004c71ac8065dc47cf5725; no voters:
I20260929 22:53:07.808537 30800 raft_consensus.cc:2781] T c25404046ba247f491276b52af370a4f P 4abdea75838e4fbd8bf5d3f863eafea7 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 22:53:07.808786 30800 raft_consensus.cc:686] T c25404046ba247f491276b52af370a4f P 4abdea75838e4fbd8bf5d3f863eafea7 [term 1 LEADER]: Becoming Leader. State: Replica: 4abdea75838e4fbd8bf5d3f863eafea7, State: Running, Role: LEADER
I20260929 22:53:07.808910 30800 consensus_queue.cc:237] T c25404046ba247f491276b52af370a4f P 4abdea75838e4fbd8bf5d3f863eafea7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1bd8776e4004c71ac8065dc47cf5725" member_type: VOTER last_known_addr { host: "127.14.147.193" port: 42211 } } peers { permanent_uuid: "28541c11b79044cf8708e09c266721fc" member_type: VOTER last_known_addr { host: "127.14.147.194" port: 46873 } } peers { permanent_uuid: "4abdea75838e4fbd8bf5d3f863eafea7" member_type: VOTER last_known_addr { host: "127.14.147.195" port: 40667 } }
I20260929 22:53:07.810007 30317 catalog_manager.cc:5495] T c25404046ba247f491276b52af370a4f P 4abdea75838e4fbd8bf5d3f863eafea7 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4abdea75838e4fbd8bf5d3f863eafea7 (127.14.147.195). New cstate: current_term: 1 leader_uuid: "4abdea75838e4fbd8bf5d3f863eafea7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1bd8776e4004c71ac8065dc47cf5725" member_type: VOTER last_known_addr { host: "127.14.147.193" port: 42211 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "28541c11b79044cf8708e09c266721fc" member_type: VOTER last_known_addr { host: "127.14.147.194" port: 46873 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4abdea75838e4fbd8bf5d3f863eafea7" member_type: VOTER last_known_addr { host: "127.14.147.195" port: 40667 } health_report { overall_health: HEALTHY } } }
I20260929 22:53:07.822515 30789 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:53:07.820179 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59642 (local address 127.14.147.195:40667)
0929 22:53:07.820244 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0929 22:53:07.820247 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 22:53:07.820255 (+ 8us) server_negotiation.cc:407] Connection header received
0929 22:53:07.820294 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 22:53:07.820295 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 22:53:07.820314 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 22:53:07.820399 (+ 85us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 22:53:07.820533 (+ 134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:53:07.821008 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:53:07.821752 (+ 744us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:53:07.821956 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:53:07.822010 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 22:53:07.822166 (+ 156us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 22:53:07.822278 (+ 112us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 22:53:07.822366 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0929 22:53:07.822381 (+ 15us) server_negotiation.cc:299] Negotiation successful
0929 22:53:07.822403 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260929 22:53:07.824481 30462 raft_consensus.cc:1261] T c25404046ba247f491276b52af370a4f P d1bd8776e4004c71ac8065dc47cf5725 [term 1 FOLLOWER]: Refusing update from remote peer 4abdea75838e4fbd8bf5d3f863eafea7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 22:53:07.824471 30596 raft_consensus.cc:1261] T c25404046ba247f491276b52af370a4f P 28541c11b79044cf8708e09c266721fc [term 1 FOLLOWER]: Refusing update from remote peer 4abdea75838e4fbd8bf5d3f863eafea7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 22:53:07.824843 30803 consensus_queue.cc:1035] T c25404046ba247f491276b52af370a4f P 4abdea75838e4fbd8bf5d3f863eafea7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1bd8776e4004c71ac8065dc47cf5725" member_type: VOTER last_known_addr { host: "127.14.147.193" port: 42211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 22:53:07.824977 30800 consensus_queue.cc:1035] T c25404046ba247f491276b52af370a4f P 4abdea75838e4fbd8bf5d3f863eafea7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "28541c11b79044cf8708e09c266721fc" member_type: VOTER last_known_addr { host: "127.14.147.194" port: 46873 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260929 22:53:07.826288 30643 tablet.cc:2369] T c25404046ba247f491276b52af370a4f P 28541c11b79044cf8708e09c266721fc: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 22:53:07.828219 30807 mvcc.cc:205] Tried to move back new op lower bound from 7334798900526215168 to 7334798900466253824. Current Snapshot: MvccSnapshot[applied={T|T < 7334798900526215168}]
I20260929 22:53:07.829872 30316 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53894:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 22:53:07.829972 30316 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53894:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 22:53:07.830757 30316 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 34cef3aa11664e048d242d3feef47476: Sending DeleteTablet for 3 replicas of tablet c25404046ba247f491276b52af370a4f
I20260929 22:53:07.831136 30576 tablet_service.cc:1514] Processing DeleteTablet for tablet c25404046ba247f491276b52af370a4f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 22:53:07 UTC) from {username='kudu', principal='kudu/127.14.147.254@KRBTEST.COM'} at 127.0.0.1:37700
I20260929 22:53:07.831136 30442 tablet_service.cc:1514] Processing DeleteTablet for tablet c25404046ba247f491276b52af370a4f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 22:53:07 UTC) from {username='kudu', principal='kudu/127.14.147.254@KRBTEST.COM'} at 127.0.0.1:53558
I20260929 22:53:07.831174 30710 tablet_service.cc:1514] Processing DeleteTablet for tablet c25404046ba247f491276b52af370a4f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 22:53:07 UTC) from {username='kudu', principal='kudu/127.14.147.254@KRBTEST.COM'} at 127.0.0.1:59636
I20260929 22:53:07.831363 30816 tablet_replica.cc:330] T c25404046ba247f491276b52af370a4f P 28541c11b79044cf8708e09c266721fc: stopping tablet replica
I20260929 22:53:07.831409 14927 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhWOCLk/build/release/bin/kudu with pid 30379
I20260929 22:53:07.831506 30816 raft_consensus.cc:2229] T c25404046ba247f491276b52af370a4f P 28541c11b79044cf8708e09c266721fc [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 22:53:07.832085 30816 raft_consensus.cc:2258] T c25404046ba247f491276b52af370a4f P 28541c11b79044cf8708e09c266721fc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 22:53:07.832535 30816 ts_tablet_manager.cc:1905] T c25404046ba247f491276b52af370a4f P 28541c11b79044cf8708e09c266721fc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 22:53:07.833771 30817 tablet_replica.cc:330] T c25404046ba247f491276b52af370a4f P 4abdea75838e4fbd8bf5d3f863eafea7: stopping tablet replica
I20260929 22:53:07.833891 30817 raft_consensus.cc:2229] T c25404046ba247f491276b52af370a4f P 4abdea75838e4fbd8bf5d3f863eafea7 [term 1 LEADER]: Raft consensus shutting down.
I20260929 22:53:07.834074 30817 raft_consensus.cc:2258] T c25404046ba247f491276b52af370a4f P 4abdea75838e4fbd8bf5d3f863eafea7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 22:53:07.834090 30816 ts_tablet_manager.cc:1918] T c25404046ba247f491276b52af370a4f P 28541c11b79044cf8708e09c266721fc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 22:53:07.834153 30816 log.cc:1196] T c25404046ba247f491276b52af370a4f P 28541c11b79044cf8708e09c266721fc: Deleting WAL directory at /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790722213368738-14927-0/minicluster-data/ts-1/wal/wals/c25404046ba247f491276b52af370a4f
I20260929 22:53:07.834517 30817 ts_tablet_manager.cc:1905] T c25404046ba247f491276b52af370a4f P 4abdea75838e4fbd8bf5d3f863eafea7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 22:53:07.834527 30816 ts_tablet_manager.cc:1939] T c25404046ba247f491276b52af370a4f P 28541c11b79044cf8708e09c266721fc: Deleting consensus metadata
I20260929 22:53:07.834998 30304 catalog_manager.cc:4841] TS 28541c11b79044cf8708e09c266721fc (127.14.147.194:46873): tablet c25404046ba247f491276b52af370a4f (table test-table [id=3335ce09a5de45cf9b202aef64333e9c]) successfully deleted
I20260929 22:53:07.835979 30817 ts_tablet_manager.cc:1918] T c25404046ba247f491276b52af370a4f P 4abdea75838e4fbd8bf5d3f863eafea7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 22:53:07.836056 30817 log.cc:1196] T c25404046ba247f491276b52af370a4f P 4abdea75838e4fbd8bf5d3f863eafea7: Deleting WAL directory at /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790722213368738-14927-0/minicluster-data/ts-2/wal/wals/c25404046ba247f491276b52af370a4f
I20260929 22:53:07.836293 30817 ts_tablet_manager.cc:1939] T c25404046ba247f491276b52af370a4f P 4abdea75838e4fbd8bf5d3f863eafea7: Deleting consensus metadata
I20260929 22:53:07.836766 30303 catalog_manager.cc:4841] TS 4abdea75838e4fbd8bf5d3f863eafea7 (127.14.147.195:40667): tablet c25404046ba247f491276b52af370a4f (table test-table [id=3335ce09a5de45cf9b202aef64333e9c]) successfully deleted
W20260929 22:53:07.837013 30304 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.14.147.193:42211 (error 108)
I20260929 22:53:07.837376 14927 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhWOCLk/build/release/bin/kudu with pid 30513
I20260929 22:53:07.837508 30788 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:53:07.837384 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.147.193:42211 (local address 127.0.0.1:53566)
0929 22:53:07.837435 (+ 51us) negotiation.cc:107] Waiting for socket to connect
0929 22:53:07.837477 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.147.193:42211: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
W20260929 22:53:07.838102 30304 catalog_manager.cc:4584] TS d1bd8776e4004c71ac8065dc47cf5725 (127.14.147.193:42211): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet c25404046ba247f491276b52af370a4f: Network error: Client connection negotiation failed: client connection to 127.14.147.193:42211: connect: Connection refused (error 111)
I20260929 22:53:07.843870 14927 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhWOCLk/build/release/bin/kudu with pid 30647
I20260929 22:53:07.848431 14927 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhWOCLk/build/release/bin/kudu with pid 30285
I20260929 22:53:27.853832 14927 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhWOCLk/build/release/bin/kudu
/tmp/dist-test-taskhWOCLk/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790722213368738-14927-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790722213368738-14927-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790722213368738-14927-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790722213368738-14927-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.147.254:35531
--webserver_interface=127.14.147.254
--webserver_port=46623
--keytab_file=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790722213368738-14927-0/krb5kdc/kudu_127.14.147.254.keytab
--principal=kudu/127.14.147.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.147.212:41357
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.147.254:35531
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--tsk_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790722213368738-14927-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790722213368738-14927-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790722213368738-14927-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 22:53:27.934846 30819 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 22:53:27.935070 30819 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 22:53:27.935093 30819 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 22:53:27.936353 30819 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260929 22:53:27.936407 30819 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260929 22:53:27.936420 30819 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260929 22:53:27.936430 30819 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 22:53:27.936442 30819 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 22:53:27.936455 30819 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 22:53:27.936465 30819 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 22:53:27.936478 30819 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 22:53:27.938047 30819 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.147.212:41357
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790722213368738-14927-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790722213368738-14927-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_private_key_password_cmd=echo foo
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.14.147.254:35531
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790722213368738-14927-0/krb5kdc/kudu_127.14.147.254.keytab
--principal=kudu/127.14.147.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.147.254:35531
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790722213368738-14927-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.14.147.254
--webserver_port=46623
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30819
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790722213368738-14927-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 21:58:54 UTC on ccycloud.quasar-xguoti.root.comops.site
build id 4681390
I20260929 22:53:27.938294 30819 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 22:53:27.938525 30819 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 22:53:27.947206 30825 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 22:53:27.947196 30828 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 22:53:27.947194 30826 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 22:53:27.947220 30819 server_base.cc:1034] running on GCE node
I20260929 22:53:27.947674 30819 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 22:53:27.947971 30819 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 22:53:27.949139 30819 hybrid_clock.cc:648] HybridClock initialized: now 1790722407949129 us; error 29 us; skew 500 ppm
Sep 29 22:53:27 dist-test-slave-98mk krb5kdc[30272](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790722407, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.147.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 22:53:27.955919 30819 init.cc:377] Logged in from keytab as kudu/127.14.147.254@KRBTEST.COM (short username kudu)
I20260929 22:53:27.956439 30819 webserver.cc:460] Webserver started at http://127.14.147.254:46623/ using document root <none> and password file <none>
I20260929 22:53:27.956702 30819 fs_manager.cc:359] Metadata directory not provided
I20260929 22:53:27.956772 30819 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 22:53:27.958258 30819 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260929 22:53:27.958920 30835 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 22:53:27.959087 30819 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260929 22:53:27.959172 30819 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790722213368738-14927-0/minicluster-data/master-0/data,/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790722213368738-14927-0/minicluster-data/master-0/wal
uuid: "34cef3aa11664e048d242d3feef47476"
format_stamp: "Formatted at 2026-09-29 22:53:07 on dist-test-slave-98mk"
server_key: "19382e3e92633e92632c2d8bb0006584"
server_key_iv: "067b6736fb0621bad8f856c90cab1be8"
server_key_version: "encryptionkey@0"
I20260929 22:53:27.959607 30819 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790722213368738-14927-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790722213368738-14927-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790722213368738-14927-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 22:53:27.968562 30819 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 22:53:27.970865 30819 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 22:53:27.970992 30819 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 22:53:27.974779 30819 rpc_server.cc:307] RPC server started. Bound to: 127.14.147.254:35531
I20260929 22:53:27.974820 30887 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.147.254:35531 every 8 connection(s)
I20260929 22:53:27.975179 30819 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790722213368738-14927-0/minicluster-data/master-0/data/info.pb
I20260929 22:53:27.976665 30888 sys_catalog.cc:263] Verifying existing consensus state
I20260929 22:53:27.977315 30888 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 34cef3aa11664e048d242d3feef47476: Bootstrap starting.
I20260929 22:53:27.979060 14927 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhWOCLk/build/release/bin/kudu as pid 30819
I20260929 22:53:27.979328 14927 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhWOCLk/build/release/bin/kudu
/tmp/dist-test-taskhWOCLk/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790722213368738-14927-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790722213368738-14927-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790722213368738-14927-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790722213368738-14927-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.147.193:42211
--local_ip_for_outbound_sockets=127.14.147.193
--tserver_master_addrs=127.14.147.254:35531
--webserver_port=44357
--webserver_interface=127.14.147.193
--keytab_file=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790722213368738-14927-0/krb5kdc/kudu_127.14.147.193.keytab
--principal=kudu/127.14.147.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.147.212:41357
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790722213368738-14927-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790722213368738-14927-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790722213368738-14927-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 22:53:27.985486 30888 log.cc:824] T 00000000000000000000000000000000 P 34cef3aa11664e048d242d3feef47476: Log is configured to *not* fsync() on all Append() calls
I20260929 22:53:27.987461 30888 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 34cef3aa11664e048d242d3feef47476: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260929 22:53:27.987743 30888 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 34cef3aa11664e048d242d3feef47476: Bootstrap complete.
I20260929 22:53:27.989820 30888 raft_consensus.cc:348] T 00000000000000000000000000000000 P 34cef3aa11664e048d242d3feef47476 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "34cef3aa11664e048d242d3feef47476" member_type: VOTER last_known_addr { host: "127.14.147.254" port: 35531 } }
I20260929 22:53:27.990049 30888 raft_consensus.cc:729] T 00000000000000000000000000000000 P 34cef3aa11664e048d242d3feef47476 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 34cef3aa11664e048d242d3feef47476, State: Initialized, Role: FOLLOWER
I20260929 22:53:27.990192 30888 consensus_queue.cc:260] T 00000000000000000000000000000000 P 34cef3aa11664e048d242d3feef47476 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "34cef3aa11664e048d242d3feef47476" member_type: VOTER last_known_addr { host: "127.14.147.254" port: 35531 } }
I20260929 22:53:27.990268 30888 raft_consensus.cc:388] T 00000000000000000000000000000000 P 34cef3aa11664e048d242d3feef47476 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 22:53:27.990319 30888 raft_consensus.cc:482] T 00000000000000000000000000000000 P 34cef3aa11664e048d242d3feef47476 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 22:53:27.990366 30888 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 34cef3aa11664e048d242d3feef47476 [term 1 FOLLOWER]: Advancing to term 2
I20260929 22:53:27.991376 30888 raft_consensus.cc:504] T 00000000000000000000000000000000 P 34cef3aa11664e048d242d3feef47476 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "34cef3aa11664e048d242d3feef47476" member_type: VOTER last_known_addr { host: "127.14.147.254" port: 35531 } }
I20260929 22:53:27.991494 30888 leader_election.cc:302] T 00000000000000000000000000000000 P 34cef3aa11664e048d242d3feef47476 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 34cef3aa11664e048d242d3feef47476; no voters:
I20260929 22:53:27.991667 30888 leader_election.cc:288] T 00000000000000000000000000000000 P 34cef3aa11664e048d242d3feef47476 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260929 22:53:27.991703 30892 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 34cef3aa11664e048d242d3feef47476 [term 2 FOLLOWER]: Leader election won for term 2
I20260929 22:53:27.991822 30892 raft_consensus.cc:686] T 00000000000000000000000000000000 P 34cef3aa11664e048d242d3feef47476 [term 2 LEADER]: Becoming Leader. State: Replica: 34cef3aa11664e048d242d3feef47476, State: Running, Role: LEADER
I20260929 22:53:27.991905 30892 consensus_queue.cc:237] T 00000000000000000000000000000000 P 34cef3aa11664e048d242d3feef47476 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "34cef3aa11664e048d242d3feef47476" member_type: VOTER last_known_addr { host: "127.14.147.254" port: 35531 } }
I20260929 22:53:27.991932 30888 sys_catalog.cc:564] T 00000000000000000000000000000000 P 34cef3aa11664e048d242d3feef47476 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 22:53:27.992223 30894 sys_catalog.cc:455] T 00000000000000000000000000000000 P 34cef3aa11664e048d242d3feef47476 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 34cef3aa11664e048d242d3feef47476. Latest consensus state: current_term: 2 leader_uuid: "34cef3aa11664e048d242d3feef47476" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "34cef3aa11664e048d242d3feef47476" member_type: VOTER last_known_addr { host: "127.14.147.254" port: 35531 } } }
I20260929 22:53:27.992210 30893 sys_catalog.cc:455] T 00000000000000000000000000000000 P 34cef3aa11664e048d242d3feef47476 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "34cef3aa11664e048d242d3feef47476" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "34cef3aa11664e048d242d3feef47476" member_type: VOTER last_known_addr { host: "127.14.147.254" port: 35531 } } }
I20260929 22:53:27.992313 30894 sys_catalog.cc:458] T 00000000000000000000000000000000 P 34cef3aa11664e048d242d3feef47476 [sys.catalog]: This master's current role is: LEADER
I20260929 22:53:27.992324 30893 sys_catalog.cc:458] T 00000000000000000000000000000000 P 34cef3aa11664e048d242d3feef47476 [sys.catalog]: This master's current role is: LEADER
I20260929 22:53:27.992497 30896 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 22:53:27.993319 30896 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 22:53:27.993429 30896 catalog_manager.cc:1259] Loaded cluster ID: 8c16912f8a7f42e883c53e5491bfb100
I20260929 22:53:27.993466 30896 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 22:53:27.994560 30896 catalog_manager.cc:1504] Loading token signing keys...
I20260929 22:53:27.994724 30896 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 34cef3aa11664e048d242d3feef47476: Loaded TSK: 0
I20260929 22:53:28.007916 30896 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 34cef3aa11664e048d242d3feef47476: Generated new TSK 1
I20260929 22:53:28.008018 30896 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260929 22:53:28.073477 30890 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 22:53:28.073714 30890 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 22:53:28.073748 30890 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 22:53:28.074997 30890 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 22:53:28.075052 30890 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 22:53:28.075088 30890 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.147.193
I20260929 22:53:28.076689 30890 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.147.212:41357
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790722213368738-14927-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790722213368738-14927-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790722213368738-14927-0/krb5kdc/kudu_127.14.147.193.keytab
--principal=kudu/127.14.147.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.147.193:42211
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790722213368738-14927-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.14.147.193
--webserver_port=44357
--tserver_master_addrs=127.14.147.254:35531
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30890
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.147.193
--log_dir=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790722213368738-14927-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 21:58:54 UTC on ccycloud.quasar-xguoti.root.comops.site
build id 4681390
I20260929 22:53:28.076949 30890 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 22:53:28.077178 30890 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 22:53:28.082026 30915 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 22:53:28.082098 30916 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 22:53:28.082024 30918 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 22:53:28.082597 30890 server_base.cc:1034] running on GCE node
I20260929 22:53:28.082811 30890 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 22:53:28.083055 30890 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 22:53:28.084205 30890 hybrid_clock.cc:648] HybridClock initialized: now 1790722408084188 us; error 33 us; skew 500 ppm
Sep 29 22:53:28 dist-test-slave-98mk krb5kdc[30272](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790722408, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.147.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 22:53:28.086642 30890 init.cc:377] Logged in from keytab as kudu/127.14.147.193@KRBTEST.COM (short username kudu)
I20260929 22:53:28.087077 30890 webserver.cc:460] Webserver started at http://127.14.147.193:44357/ using document root <none> and password file <none>
I20260929 22:53:28.087296 30890 fs_manager.cc:359] Metadata directory not provided
I20260929 22:53:28.087354 30890 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 22:53:28.088766 30890 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260929 22:53:28.089483 30925 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 22:53:28.089663 30890 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260929 22:53:28.089751 30890 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790722213368738-14927-0/minicluster-data/ts-0/data,/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790722213368738-14927-0/minicluster-data/ts-0/wal
uuid: "d1bd8776e4004c71ac8065dc47cf5725"
format_stamp: "Formatted at 2026-09-29 22:53:07 on dist-test-slave-98mk"
server_key: "73d4082730f9488ee6a939b600ad086f"
server_key_iv: "bfc24ed012d73abd308ad29d5d8007c3"
server_key_version: "encryptionkey@0"
I20260929 22:53:28.090164 30890 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790722213368738-14927-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790722213368738-14927-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790722213368738-14927-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 22:53:28.107671 30890 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 22:53:28.109767 30890 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 22:53:28.109905 30890 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 22:53:28.110121 30890 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 22:53:28.110541 30932 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260929 22:53:28.112505 30890 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260929 22:53:28.112566 30890 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260929 22:53:28.112604 30890 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260929 22:53:28.113235 30890 ts_tablet_manager.cc:610] Registered 1 tablets
I20260929 22:53:28.113269 30890 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20260929 22:53:28.113356 30932 tablet_bootstrap.cc:492] T c25404046ba247f491276b52af370a4f P d1bd8776e4004c71ac8065dc47cf5725: Bootstrap starting.
I20260929 22:53:28.119837 30890 rpc_server.cc:307] RPC server started. Bound to: 127.14.147.193:42211
I20260929 22:53:28.120019 31040 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.147.193:42211 every 8 connection(s)
I20260929 22:53:28.120275 30890 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790722213368738-14927-0/minicluster-data/ts-0/data/info.pb
I20260929 22:53:28.124611 14927 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhWOCLk/build/release/bin/kudu as pid 30890
I20260929 22:53:28.124925 14927 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhWOCLk/build/release/bin/kudu
/tmp/dist-test-taskhWOCLk/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790722213368738-14927-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790722213368738-14927-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790722213368738-14927-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790722213368738-14927-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.147.194:46873
--local_ip_for_outbound_sockets=127.14.147.194
--tserver_master_addrs=127.14.147.254:35531
--webserver_port=42367
--webserver_interface=127.14.147.194
--keytab_file=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790722213368738-14927-0/krb5kdc/kudu_127.14.147.194.keytab
--principal=kudu/127.14.147.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.147.212:41357
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790722213368738-14927-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790722213368738-14927-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790722213368738-14927-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 22:53:28.124913 30932 log.cc:824] T c25404046ba247f491276b52af370a4f P d1bd8776e4004c71ac8065dc47cf5725: Log is configured to *not* fsync() on all Append() calls
I20260929 22:53:28.127549 30932 tablet_bootstrap.cc:492] T c25404046ba247f491276b52af370a4f P d1bd8776e4004c71ac8065dc47cf5725: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260929 22:53:28.127859 30932 tablet_bootstrap.cc:492] T c25404046ba247f491276b52af370a4f P d1bd8776e4004c71ac8065dc47cf5725: Bootstrap complete.
I20260929 22:53:28.128981 30932 ts_tablet_manager.cc:1397] T c25404046ba247f491276b52af370a4f P d1bd8776e4004c71ac8065dc47cf5725: Time spent bootstrapping tablet: real 0.016s user 0.010s sys 0.005s
Sep 29 22:53:28 dist-test-slave-98mk krb5kdc[30272](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790722408, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.147.193@KRBTEST.COM for kudu/127.14.147.254@KRBTEST.COM
I20260929 22:53:28.130641 30932 raft_consensus.cc:348] T c25404046ba247f491276b52af370a4f P d1bd8776e4004c71ac8065dc47cf5725 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1bd8776e4004c71ac8065dc47cf5725" member_type: VOTER last_known_addr { host: "127.14.147.193" port: 42211 } } peers { permanent_uuid: "28541c11b79044cf8708e09c266721fc" member_type: VOTER last_known_addr { host: "127.14.147.194" port: 46873 } } peers { permanent_uuid: "4abdea75838e4fbd8bf5d3f863eafea7" member_type: VOTER last_known_addr { host: "127.14.147.195" port: 40667 } }
I20260929 22:53:28.130965 30932 raft_consensus.cc:729] T c25404046ba247f491276b52af370a4f P d1bd8776e4004c71ac8065dc47cf5725 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: d1bd8776e4004c71ac8065dc47cf5725, State: Initialized, Role: FOLLOWER
I20260929 22:53:28.131127 30932 consensus_queue.cc:260] T c25404046ba247f491276b52af370a4f P d1bd8776e4004c71ac8065dc47cf5725 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1bd8776e4004c71ac8065dc47cf5725" member_type: VOTER last_known_addr { host: "127.14.147.193" port: 42211 } } peers { permanent_uuid: "28541c11b79044cf8708e09c266721fc" member_type: VOTER last_known_addr { host: "127.14.147.194" port: 46873 } } peers { permanent_uuid: "4abdea75838e4fbd8bf5d3f863eafea7" member_type: VOTER last_known_addr { host: "127.14.147.195" port: 40667 } }
I20260929 22:53:28.131397 30932 ts_tablet_manager.cc:1428] T c25404046ba247f491276b52af370a4f P d1bd8776e4004c71ac8065dc47cf5725: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20260929 22:53:28.135577 31043 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:53:28.120986 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.147.254:35531 (local address 127.14.147.193:60893)
0929 22:53:28.123557 (+ 2571us) negotiation.cc:107] Waiting for socket to connect
0929 22:53:28.123605 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0929 22:53:28.124099 (+ 494us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 22:53:28.124791 (+ 692us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 22:53:28.124795 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 22:53:28.125172 (+ 377us) client_negotiation.cc:189] Negotiated authn=SASL
0929 22:53:28.125452 (+ 280us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 22:53:28.125459 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 22:53:28.127152 (+ 1693us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 22:53:28.127153 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 22:53:28.128282 (+ 1129us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 22:53:28.128284 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 22:53:28.128868 (+ 584us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 22:53:28.128869 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 22:53:28.129008 (+ 139us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 22:53:28.129608 (+ 600us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 22:53:28.129618 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0929 22:53:28.131324 (+ 1706us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 22:53:28.133650 (+ 2326us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 22:53:28.133661 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 22:53:28.133663 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 22:53:28.133986 (+ 323us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 22:53:28.134296 (+ 310us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 22:53:28.134297 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 22:53:28.134297 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 22:53:28.134428 (+ 131us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 22:53:28.134892 (+ 464us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 22:53:28.134898 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 22:53:28.135050 (+ 152us) client_negotiation.cc:769] Sending connection context
0929 22:53:28.135167 (+ 117us) client_negotiation.cc:240] Negotiation successful
0929 22:53:28.135335 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2519,"thread_start_us":96,"threads_started":1}
I20260929 22:53:28.135627 31044 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:53:28.121067 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.147.193:60893 (local address 127.14.147.254:35531)
0929 22:53:28.124282 (+ 3215us) server_negotiation.cc:206] Beginning negotiation
0929 22:53:28.124290 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 22:53:28.124318 (+ 28us) server_negotiation.cc:407] Connection header received
0929 22:53:28.124446 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 22:53:28.124453 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 22:53:28.124510 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 22:53:28.124639 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0929 22:53:28.125602 (+ 963us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:53:28.126973 (+ 1371us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:53:28.128365 (+ 1392us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:53:28.128775 (+ 410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:53:28.128871 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 22:53:28.131473 (+ 2602us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 22:53:28.131477 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 22:53:28.131478 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 22:53:28.131491 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0929 22:53:28.133541 (+ 2050us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 22:53:28.134085 (+ 544us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 22:53:28.134089 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 22:53:28.134089 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 22:53:28.134222 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 22:53:28.134497 (+ 275us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 22:53:28.134498 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 22:53:28.134499 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 22:53:28.134812 (+ 313us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 22:53:28.134902 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0929 22:53:28.135242 (+ 340us) server_negotiation.cc:299] Negotiation successful
0929 22:53:28.135400 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3166,"thread_start_us":66,"threads_started":1}
I20260929 22:53:28.136250 31041 heartbeater.cc:347] Connected to a master server at 127.14.147.254:35531
I20260929 22:53:28.136356 31041 heartbeater.cc:464] Registering TS with master...
I20260929 22:53:28.136552 31041 heartbeater.cc:511] Master 127.14.147.254:35531 requested a full tablet report, sending...
I20260929 22:53:28.137367 30852 ts_manager.cc:194] Registered new tserver with Master: d1bd8776e4004c71ac8065dc47cf5725 (127.14.147.193:42211)
I20260929 22:53:28.138564 30852 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.147.193@KRBTEST.COM'} at 127.14.147.193:60893
I20260929 22:53:28.139575 31041 heartbeater.cc:503] Master 127.14.147.254:35531 was elected leader, sending a full tablet report...
Sep 29 22:53:28 dist-test-slave-98mk krb5kdc[30272](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790722407, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.147.254@KRBTEST.COM for kudu/127.14.147.193@KRBTEST.COM
I20260929 22:53:28.147832 31048 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:53:28.138022 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.147.193:42211 (local address 127.0.0.1:38204)
0929 22:53:28.138853 (+ 831us) negotiation.cc:107] Waiting for socket to connect
0929 22:53:28.138886 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0929 22:53:28.138949 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 22:53:28.139215 (+ 266us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 22:53:28.139220 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 22:53:28.139567 (+ 347us) client_negotiation.cc:189] Negotiated authn=SASL
0929 22:53:28.139640 (+ 73us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 22:53:28.139643 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 22:53:28.140837 (+ 1194us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 22:53:28.140838 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 22:53:28.141861 (+ 1023us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 22:53:28.141864 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 22:53:28.142460 (+ 596us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 22:53:28.142461 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 22:53:28.142539 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 22:53:28.143113 (+ 574us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 22:53:28.143118 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0929 22:53:28.144603 (+ 1485us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 22:53:28.146660 (+ 2057us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 22:53:28.146668 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 22:53:28.146670 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 22:53:28.146920 (+ 250us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 22:53:28.147142 (+ 222us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 22:53:28.147142 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 22:53:28.147143 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 22:53:28.147201 (+ 58us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 22:53:28.147518 (+ 317us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 22:53:28.147524 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 22:53:28.147574 (+ 50us) client_negotiation.cc:769] Sending connection context
0929 22:53:28.147647 (+ 73us) client_negotiation.cc:240] Negotiation successful
0929 22:53:28.147757 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":804,"thread_start_us":112,"threads_started":1}
I20260929 22:53:28.147882 31049 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:53:28.138310 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38204 (local address 127.14.147.193:42211)
0929 22:53:28.139068 (+ 758us) server_negotiation.cc:206] Beginning negotiation
0929 22:53:28.139072 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 22:53:28.139080 (+ 8us) server_negotiation.cc:407] Connection header received
0929 22:53:28.139120 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 22:53:28.139122 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 22:53:28.139135 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 22:53:28.139188 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0929 22:53:28.139766 (+ 578us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:53:28.140738 (+ 972us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:53:28.141953 (+ 1215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:53:28.142381 (+ 428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:53:28.142457 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 22:53:28.144697 (+ 2240us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 22:53:28.144709 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 22:53:28.144709 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 22:53:28.144718 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0929 22:53:28.146532 (+ 1814us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 22:53:28.146996 (+ 464us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 22:53:28.147000 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 22:53:28.147000 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 22:53:28.147066 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 22:53:28.147249 (+ 183us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 22:53:28.147250 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 22:53:28.147250 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 22:53:28.147457 (+ 207us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 22:53:28.147544 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0929 22:53:28.147693 (+ 149us) server_negotiation.cc:299] Negotiation successful
0929 22:53:28.147778 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":727,"thread_start_us":85,"threads_started":1}
I20260929 22:53:28.148296 30975 tablet_service.cc:1514] Processing DeleteTablet for tablet c25404046ba247f491276b52af370a4f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 22:53:07 UTC) from {username='kudu', principal='kudu/127.14.147.254@KRBTEST.COM'} at 127.0.0.1:38204
I20260929 22:53:28.148913 31050 tablet_replica.cc:330] stopping tablet replica
I20260929 22:53:28.149021 31050 raft_consensus.cc:2229] T c25404046ba247f491276b52af370a4f P d1bd8776e4004c71ac8065dc47cf5725 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 22:53:28.149093 31050 raft_consensus.cc:2258] T c25404046ba247f491276b52af370a4f P d1bd8776e4004c71ac8065dc47cf5725 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 22:53:28.149407 31050 ts_tablet_manager.cc:1905] T c25404046ba247f491276b52af370a4f P d1bd8776e4004c71ac8065dc47cf5725: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 22:53:28.151449 31050 ts_tablet_manager.cc:1918] T c25404046ba247f491276b52af370a4f P d1bd8776e4004c71ac8065dc47cf5725: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 22:53:28.151528 31050 log.cc:1196] T c25404046ba247f491276b52af370a4f P d1bd8776e4004c71ac8065dc47cf5725: Deleting WAL directory at /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790722213368738-14927-0/minicluster-data/ts-0/wal/wals/c25404046ba247f491276b52af370a4f
I20260929 22:53:28.151715 31050 ts_tablet_manager.cc:1939] T c25404046ba247f491276b52af370a4f P d1bd8776e4004c71ac8065dc47cf5725: Deleting consensus metadata
I20260929 22:53:28.152215 30839 catalog_manager.cc:4841] TS d1bd8776e4004c71ac8065dc47cf5725 (127.14.147.193:42211): tablet c25404046ba247f491276b52af370a4f (table test-table [id=3335ce09a5de45cf9b202aef64333e9c]) successfully deleted
W20260929 22:53:28.214146 31046 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 22:53:28.214359 31046 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 22:53:28.214383 31046 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 22:53:28.215513 31046 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 22:53:28.215549 31046 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 22:53:28.215574 31046 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.147.194
I20260929 22:53:28.217103 31046 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.147.212:41357
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790722213368738-14927-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790722213368738-14927-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790722213368738-14927-0/krb5kdc/kudu_127.14.147.194.keytab
--principal=kudu/127.14.147.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.147.194:46873
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790722213368738-14927-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.14.147.194
--webserver_port=42367
--tserver_master_addrs=127.14.147.254:35531
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31046
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.147.194
--log_dir=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790722213368738-14927-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 21:58:54 UTC on ccycloud.quasar-xguoti.root.comops.site
build id 4681390
I20260929 22:53:28.217343 31046 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 22:53:28.217581 31046 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 22:53:28.222354 31055 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 22:53:28.222332 31056 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 22:53:28.222350 31058 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 22:53:28.222725 31046 server_base.cc:1034] running on GCE node
I20260929 22:53:28.222922 31046 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 22:53:28.223158 31046 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 22:53:28.224306 31046 hybrid_clock.cc:648] HybridClock initialized: now 1790722408224286 us; error 37 us; skew 500 ppm
Sep 29 22:53:28 dist-test-slave-98mk krb5kdc[30272](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790722408, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.147.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 22:53:28.226761 31046 init.cc:377] Logged in from keytab as kudu/127.14.147.194@KRBTEST.COM (short username kudu)
I20260929 22:53:28.227205 31046 webserver.cc:460] Webserver started at http://127.14.147.194:42367/ using document root <none> and password file <none>
I20260929 22:53:28.227437 31046 fs_manager.cc:359] Metadata directory not provided
I20260929 22:53:28.227514 31046 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 22:53:28.228880 31046 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260929 22:53:28.229800 31065 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 22:53:28.229975 31046 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260929 22:53:28.230063 31046 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790722213368738-14927-0/minicluster-data/ts-1/data,/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790722213368738-14927-0/minicluster-data/ts-1/wal
uuid: "28541c11b79044cf8708e09c266721fc"
format_stamp: "Formatted at 2026-09-29 22:53:07 on dist-test-slave-98mk"
server_key: "cfc2dc837f7660f34a1a910a5b5658bf"
server_key_iv: "f2f98582a6acbe96e50692ba98c1f05b"
server_key_version: "encryptionkey@0"
I20260929 22:53:28.230427 31046 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790722213368738-14927-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790722213368738-14927-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790722213368738-14927-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 22:53:28.240350 31046 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 22:53:28.242483 31046 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 22:53:28.242626 31046 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 22:53:28.242851 31046 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 22:53:28.243147 31046 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 22:53:28.243180 31046 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 22:53:28.243208 31046 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 22:53:28.243225 31046 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 22:53:28.250108 31046 rpc_server.cc:307] RPC server started. Bound to: 127.14.147.194:46873
I20260929 22:53:28.250145 31178 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.147.194:46873 every 8 connection(s)
I20260929 22:53:28.250478 31046 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790722213368738-14927-0/minicluster-data/ts-1/data/info.pb
I20260929 22:53:28.252385 14927 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhWOCLk/build/release/bin/kudu as pid 31046
I20260929 22:53:28.252712 14927 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhWOCLk/build/release/bin/kudu
/tmp/dist-test-taskhWOCLk/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790722213368738-14927-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790722213368738-14927-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790722213368738-14927-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790722213368738-14927-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.147.195:40667
--local_ip_for_outbound_sockets=127.14.147.195
--tserver_master_addrs=127.14.147.254:35531
--webserver_port=40869
--webserver_interface=127.14.147.195
--keytab_file=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790722213368738-14927-0/krb5kdc/kudu_127.14.147.195.keytab
--principal=kudu/127.14.147.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.147.212:41357
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790722213368738-14927-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790722213368738-14927-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790722213368738-14927-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 29 22:53:28 dist-test-slave-98mk krb5kdc[30272](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790722408, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.147.194@KRBTEST.COM for kudu/127.14.147.254@KRBTEST.COM
I20260929 22:53:28.260223 31044 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:53:28.251247 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.147.194:59639 (local address 127.14.147.254:35531)
0929 22:53:28.251297 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0929 22:53:28.251299 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 22:53:28.251514 (+ 215us) server_negotiation.cc:407] Connection header received
0929 22:53:28.252088 (+ 574us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 22:53:28.252089 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 22:53:28.252104 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 22:53:28.252185 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0929 22:53:28.253441 (+ 1256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:53:28.253950 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:53:28.254994 (+ 1044us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:53:28.255191 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:53:28.255247 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 22:53:28.257481 (+ 2234us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 22:53:28.257482 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 22:53:28.257483 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 22:53:28.257492 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0929 22:53:28.258809 (+ 1317us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 22:53:28.259222 (+ 413us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 22:53:28.259223 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 22:53:28.259224 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 22:53:28.259278 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 22:53:28.259556 (+ 278us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 22:53:28.259557 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 22:53:28.259557 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 22:53:28.259681 (+ 124us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 22:53:28.259737 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0929 22:53:28.260014 (+ 277us) server_negotiation.cc:299] Negotiation successful
0929 22:53:28.260124 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260929 22:53:28.260357 31181 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:53:28.251193 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.147.254:35531 (local address 127.14.147.194:59639)
0929 22:53:28.251375 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0929 22:53:28.251438 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0929 22:53:28.251969 (+ 531us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 22:53:28.252267 (+ 298us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 22:53:28.252272 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 22:53:28.253043 (+ 771us) client_negotiation.cc:189] Negotiated authn=SASL
0929 22:53:28.253334 (+ 291us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 22:53:28.253341 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 22:53:28.254084 (+ 743us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 22:53:28.254085 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 22:53:28.254895 (+ 810us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 22:53:28.254898 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 22:53:28.255266 (+ 368us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 22:53:28.255267 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 22:53:28.255387 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 22:53:28.256020 (+ 633us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 22:53:28.256030 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0929 22:53:28.257382 (+ 1352us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 22:53:28.258905 (+ 1523us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 22:53:28.258915 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 22:53:28.258918 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 22:53:28.259150 (+ 232us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 22:53:28.259333 (+ 183us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 22:53:28.259334 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 22:53:28.259334 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 22:53:28.259489 (+ 155us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 22:53:28.259736 (+ 247us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 22:53:28.259743 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 22:53:28.259886 (+ 143us) client_negotiation.cc:769] Sending connection context
0929 22:53:28.259970 (+ 84us) client_negotiation.cc:240] Negotiation successful
0929 22:53:28.260137 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":136,"thread_start_us":88,"threads_started":1}
I20260929 22:53:28.260793 31179 heartbeater.cc:347] Connected to a master server at 127.14.147.254:35531
I20260929 22:53:28.260885 31179 heartbeater.cc:464] Registering TS with master...
I20260929 22:53:28.261065 31179 heartbeater.cc:511] Master 127.14.147.254:35531 requested a full tablet report, sending...
I20260929 22:53:28.261430 30852 ts_manager.cc:194] Registered new tserver with Master: 28541c11b79044cf8708e09c266721fc (127.14.147.194:46873)
I20260929 22:53:28.261818 30852 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.147.194@KRBTEST.COM'} at 127.14.147.194:59639
W20260929 22:53:28.333216 31182 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 22:53:28.333441 31182 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 22:53:28.333467 31182 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 22:53:28.334738 31182 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 22:53:28.334794 31182 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 22:53:28.334832 31182 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.147.195
I20260929 22:53:28.336426 31182 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.147.212:41357
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790722213368738-14927-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790722213368738-14927-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790722213368738-14927-0/krb5kdc/kudu_127.14.147.195.keytab
--principal=kudu/127.14.147.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.147.195:40667
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790722213368738-14927-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.14.147.195
--webserver_port=40869
--tserver_master_addrs=127.14.147.254:35531
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31182
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.147.195
--log_dir=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790722213368738-14927-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 21:58:54 UTC on ccycloud.quasar-xguoti.root.comops.site
build id 4681390
I20260929 22:53:28.336663 31182 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 22:53:28.336884 31182 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 22:53:28.341747 31187 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 22:53:28.341740 31190 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 22:53:28.341740 31188 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 22:53:28.341938 31182 server_base.cc:1034] running on GCE node
I20260929 22:53:28.342238 31182 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 22:53:28.342479 31182 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 22:53:28.343626 31182 hybrid_clock.cc:648] HybridClock initialized: now 1790722408343610 us; error 29 us; skew 500 ppm
Sep 29 22:53:28 dist-test-slave-98mk krb5kdc[30272](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790722408, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.147.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 22:53:28.346155 31182 init.cc:377] Logged in from keytab as kudu/127.14.147.195@KRBTEST.COM (short username kudu)
I20260929 22:53:28.346977 31182 webserver.cc:460] Webserver started at http://127.14.147.195:40869/ using document root <none> and password file <none>
I20260929 22:53:28.347316 31182 fs_manager.cc:359] Metadata directory not provided
I20260929 22:53:28.347437 31182 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 22:53:28.348960 31182 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260929 22:53:28.349591 31197 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 22:53:28.349784 31182 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260929 22:53:28.349871 31182 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790722213368738-14927-0/minicluster-data/ts-2/data,/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790722213368738-14927-0/minicluster-data/ts-2/wal
uuid: "4abdea75838e4fbd8bf5d3f863eafea7"
format_stamp: "Formatted at 2026-09-29 22:53:07 on dist-test-slave-98mk"
server_key: "1da1349444d8dccc82d5d454eec06bef"
server_key_iv: "5cdc9391c1f1db9bf81ed2981898a8d6"
server_key_version: "encryptionkey@0"
I20260929 22:53:28.350240 31182 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790722213368738-14927-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790722213368738-14927-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790722213368738-14927-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 22:53:28.361645 31182 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 22:53:28.364349 31182 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 22:53:28.364501 31182 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 22:53:28.364729 31182 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 22:53:28.365038 31182 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 22:53:28.365070 31182 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 22:53:28.365099 31182 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 22:53:28.365123 31182 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 22:53:28.371812 31182 rpc_server.cc:307] RPC server started. Bound to: 127.14.147.195:40667
I20260929 22:53:28.371919 31310 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.147.195:40667 every 8 connection(s)
I20260929 22:53:28.372234 31182 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790722213368738-14927-0/minicluster-data/ts-2/data/info.pb
Sep 29 22:53:28 dist-test-slave-98mk krb5kdc[30272](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790722408, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.147.195@KRBTEST.COM for kudu/127.14.147.254@KRBTEST.COM
I20260929 22:53:28.378041 14927 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhWOCLk/build/release/bin/kudu as pid 31182
I20260929 22:53:28.381361 31044 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:53:28.372897 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.147.195:56667 (local address 127.14.147.254:35531)
0929 22:53:28.372944 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0929 22:53:28.372945 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0929 22:53:28.373139 (+ 194us) server_negotiation.cc:407] Connection header received
0929 22:53:28.373621 (+ 482us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 22:53:28.373623 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 22:53:28.373635 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 22:53:28.373686 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0929 22:53:28.374595 (+ 909us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:53:28.375082 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:53:28.376062 (+ 980us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:53:28.376249 (+ 187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:53:28.376300 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 22:53:28.378539 (+ 2239us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 22:53:28.378540 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 22:53:28.378541 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 22:53:28.378549 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0929 22:53:28.379972 (+ 1423us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 22:53:28.380442 (+ 470us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 22:53:28.380443 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 22:53:28.380444 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 22:53:28.380519 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 22:53:28.380736 (+ 217us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 22:53:28.380736 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 22:53:28.380737 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 22:53:28.380861 (+ 124us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 22:53:28.380919 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0929 22:53:28.381220 (+ 301us) server_negotiation.cc:299] Negotiation successful
0929 22:53:28.381288 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260929 22:53:28.381625 31313 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:53:28.372890 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.147.254:35531 (local address 127.14.147.195:56667)
0929 22:53:28.373025 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0929 22:53:28.373061 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0929 22:53:28.373536 (+ 475us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 22:53:28.373740 (+ 204us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 22:53:28.373743 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 22:53:28.374315 (+ 572us) client_negotiation.cc:189] Negotiated authn=SASL
0929 22:53:28.374519 (+ 204us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 22:53:28.374524 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 22:53:28.375162 (+ 638us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 22:53:28.375163 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 22:53:28.375994 (+ 831us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 22:53:28.375997 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 22:53:28.376316 (+ 319us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 22:53:28.376316 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 22:53:28.376416 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 22:53:28.376948 (+ 532us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 22:53:28.376957 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0929 22:53:28.378416 (+ 1459us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 22:53:28.380067 (+ 1651us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 22:53:28.380078 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 22:53:28.380081 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 22:53:28.380377 (+ 296us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 22:53:28.380581 (+ 204us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 22:53:28.380582 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 22:53:28.380582 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 22:53:28.380686 (+ 104us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 22:53:28.380917 (+ 231us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 22:53:28.380920 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 22:53:28.381085 (+ 165us) client_negotiation.cc:769] Sending connection context
0929 22:53:28.381193 (+ 108us) client_negotiation.cc:240] Negotiation successful
0929 22:53:28.381372 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":100,"thread_start_us":73,"threads_started":1}
I20260929 22:53:28.382091 31311 heartbeater.cc:347] Connected to a master server at 127.14.147.254:35531
I20260929 22:53:28.382211 31311 heartbeater.cc:464] Registering TS with master...
I20260929 22:53:28.382422 31311 heartbeater.cc:511] Master 127.14.147.254:35531 requested a full tablet report, sending...
I20260929 22:53:28.382853 30852 ts_manager.cc:194] Registered new tserver with Master: 4abdea75838e4fbd8bf5d3f863eafea7 (127.14.147.195:40667)
I20260929 22:53:28.383335 30852 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.147.195@KRBTEST.COM'} at 127.14.147.195:56667
I20260929 22:53:28.385288 31315 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:53:28.378559 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49296 (local address 127.14.147.254:35531)
0929 22:53:28.378663 (+ 104us) server_negotiation.cc:206] Beginning negotiation
0929 22:53:28.378665 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 22:53:28.378718 (+ 53us) server_negotiation.cc:407] Connection header received
0929 22:53:28.378785 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 22:53:28.378786 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 22:53:28.378798 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 22:53:28.378875 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0929 22:53:28.379596 (+ 721us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:53:28.380121 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:53:28.381031 (+ 910us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:53:28.381321 (+ 290us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:53:28.381387 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 22:53:28.382971 (+ 1584us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 22:53:28.382972 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 22:53:28.382972 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 22:53:28.382979 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0929 22:53:28.384425 (+ 1446us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 22:53:28.384733 (+ 308us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 22:53:28.384733 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 22:53:28.384733 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 22:53:28.384780 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 22:53:28.384914 (+ 134us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 22:53:28.384914 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 22:53:28.384915 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 22:53:28.385017 (+ 102us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 22:53:28.385139 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0929 22:53:28.385165 (+ 26us) server_negotiation.cc:299] Negotiation successful
0929 22:53:28.385224 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85,"thread_start_us":49,"threads_started":1}
I20260929 22:53:28.385895 14927 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 22:53:28.388347 31315 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:53:28.386198 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49312 (local address 127.14.147.254:35531)
0929 22:53:28.386241 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0929 22:53:28.386242 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0929 22:53:28.386336 (+ 94us) server_negotiation.cc:407] Connection header received
0929 22:53:28.386388 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 22:53:28.386389 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 22:53:28.386418 (+ 29us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 22:53:28.386475 (+ 57us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 22:53:28.386562 (+ 87us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:53:28.387025 (+ 463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:53:28.387792 (+ 767us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:53:28.387989 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:53:28.388031 (+ 42us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 22:53:28.388132 (+ 101us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 22:53:28.388173 (+ 41us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_AUTHENTICATION_TOKEN: Not authorized: token expired
0929 22:53:28.388302 (+ 129us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:49312: token expired
Metrics: {"server-negotiator.queue_time_us":22}
W20260929 22:53:28.388394 31315 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:49312: token expired
W20260929 22:53:28.388453 30779 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.14.147.254:35531)
I20260929 22:53:28.394083 31315 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:53:28.388677 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49316 (local address 127.14.147.254:35531)
0929 22:53:28.388711 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0929 22:53:28.388713 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 22:53:28.388747 (+ 34us) server_negotiation.cc:407] Connection header received
0929 22:53:28.388814 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 22:53:28.388815 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 22:53:28.388824 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 22:53:28.388866 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0929 22:53:28.389318 (+ 452us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:53:28.389769 (+ 451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:53:28.390427 (+ 658us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:53:28.390619 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:53:28.390670 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 22:53:28.391805 (+ 1135us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 22:53:28.391806 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 22:53:28.391806 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 22:53:28.391814 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0929 22:53:28.393161 (+ 1347us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 22:53:28.393485 (+ 324us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 22:53:28.393485 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 22:53:28.393486 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 22:53:28.393534 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 22:53:28.393674 (+ 140us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 22:53:28.393674 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 22:53:28.393674 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 22:53:28.393789 (+ 115us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 22:53:28.393860 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0929 22:53:28.393945 (+ 85us) server_negotiation.cc:299] Negotiation successful
0929 22:53:28.394016 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260929 22:53:28.395218 30852 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49316:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 22:53:28.396800 30852 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 22:53:28.398828 30975 tablet_service.cc:1467] Processing CreateTablet for tablet 3658e3215a6947cfb6b38dae0c9f6cdc (DEFAULT_TABLE table=test-table [id=035d5e2be18344649f2c373426f77aab]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 22:53:28.399060 30975 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3658e3215a6947cfb6b38dae0c9f6cdc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 22:53:28.400632 30932 tablet_bootstrap.cc:492] T 3658e3215a6947cfb6b38dae0c9f6cdc P d1bd8776e4004c71ac8065dc47cf5725: Bootstrap starting.
I20260929 22:53:28.400924 30932 tablet_bootstrap.cc:654] T 3658e3215a6947cfb6b38dae0c9f6cdc P d1bd8776e4004c71ac8065dc47cf5725: Neither blocks nor log segments found. Creating new log.
I20260929 22:53:28.401409 30932 tablet_bootstrap.cc:492] T 3658e3215a6947cfb6b38dae0c9f6cdc P d1bd8776e4004c71ac8065dc47cf5725: No bootstrap required, opened a new log
I20260929 22:53:28.401454 30932 ts_tablet_manager.cc:1397] T 3658e3215a6947cfb6b38dae0c9f6cdc P d1bd8776e4004c71ac8065dc47cf5725: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260929 22:53:28.401822 30932 raft_consensus.cc:348] T 3658e3215a6947cfb6b38dae0c9f6cdc P d1bd8776e4004c71ac8065dc47cf5725 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28541c11b79044cf8708e09c266721fc" member_type: VOTER last_known_addr { host: "127.14.147.194" port: 46873 } } peers { permanent_uuid: "4abdea75838e4fbd8bf5d3f863eafea7" member_type: VOTER last_known_addr { host: "127.14.147.195" port: 40667 } } peers { permanent_uuid: "d1bd8776e4004c71ac8065dc47cf5725" member_type: VOTER last_known_addr { host: "127.14.147.193" port: 42211 } }
I20260929 22:53:28.401893 30932 raft_consensus.cc:374] T 3658e3215a6947cfb6b38dae0c9f6cdc P d1bd8776e4004c71ac8065dc47cf5725 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 22:53:28.401953 30932 raft_consensus.cc:729] T 3658e3215a6947cfb6b38dae0c9f6cdc P d1bd8776e4004c71ac8065dc47cf5725 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d1bd8776e4004c71ac8065dc47cf5725, State: Initialized, Role: FOLLOWER
I20260929 22:53:28.401995 30932 consensus_queue.cc:260] T 3658e3215a6947cfb6b38dae0c9f6cdc P d1bd8776e4004c71ac8065dc47cf5725 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28541c11b79044cf8708e09c266721fc" member_type: VOTER last_known_addr { host: "127.14.147.194" port: 46873 } } peers { permanent_uuid: "4abdea75838e4fbd8bf5d3f863eafea7" member_type: VOTER last_known_addr { host: "127.14.147.195" port: 40667 } } peers { permanent_uuid: "d1bd8776e4004c71ac8065dc47cf5725" member_type: VOTER last_known_addr { host: "127.14.147.193" port: 42211 } }
I20260929 22:53:28.402084 30932 ts_tablet_manager.cc:1428] T 3658e3215a6947cfb6b38dae0c9f6cdc P d1bd8776e4004c71ac8065dc47cf5725: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260929 22:53:28.402436 31317 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:53:28.398558 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.147.195:40667 (local address 127.0.0.1:44394)
0929 22:53:28.398812 (+ 254us) negotiation.cc:107] Waiting for socket to connect
0929 22:53:28.398822 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0929 22:53:28.398886 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 22:53:28.399148 (+ 262us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 22:53:28.399149 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 22:53:28.399151 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 22:53:28.399241 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 22:53:28.399243 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 22:53:28.400154 (+ 911us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 22:53:28.400155 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 22:53:28.401325 (+ 1170us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 22:53:28.401328 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 22:53:28.402151 (+ 823us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 22:53:28.402152 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 22:53:28.402219 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 22:53:28.402252 (+ 33us) client_negotiation.cc:769] Sending connection context
0929 22:53:28.402308 (+ 56us) client_negotiation.cc:240] Negotiation successful
0929 22:53:28.402361 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":235,"thread_start_us":113,"threads_started":1}
I20260929 22:53:28.402617 31319 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:53:28.398821 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44394 (local address 127.14.147.195:40667)
0929 22:53:28.399006 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0929 22:53:28.399010 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 22:53:28.399018 (+ 8us) server_negotiation.cc:407] Connection header received
0929 22:53:28.399055 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 22:53:28.399058 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 22:53:28.399071 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 22:53:28.399151 (+ 80us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 22:53:28.399306 (+ 155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:53:28.400072 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:53:28.401422 (+ 1350us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:53:28.402054 (+ 632us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:53:28.402126 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 22:53:28.402190 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0929 22:53:28.402388 (+ 198us) server_negotiation.cc:299] Negotiation successful
0929 22:53:28.402434 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":154,"thread_start_us":79,"threads_started":1}
I20260929 22:53:28.403280 31048 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:53:28.398501 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.147.194:46873 (local address 127.0.0.1:44570)
0929 22:53:28.398547 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0929 22:53:28.398558 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0929 22:53:28.398595 (+ 37us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 22:53:28.399037 (+ 442us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 22:53:28.399038 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 22:53:28.399040 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 22:53:28.399096 (+ 56us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 22:53:28.399097 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 22:53:28.400387 (+ 1290us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 22:53:28.400388 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 22:53:28.401416 (+ 1028us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 22:53:28.401418 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 22:53:28.403062 (+ 1644us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 22:53:28.403063 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 22:53:28.403147 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 22:53:28.403150 (+ 3us) client_negotiation.cc:769] Sending connection context
0929 22:53:28.403191 (+ 41us) client_negotiation.cc:240] Negotiation successful
0929 22:53:28.403232 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":27}
I20260929 22:53:28.403446 31318 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:53:28.398644 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44570 (local address 127.14.147.194:46873)
0929 22:53:28.398819 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0929 22:53:28.398823 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 22:53:28.398834 (+ 11us) server_negotiation.cc:407] Connection header received
0929 22:53:28.398873 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 22:53:28.398875 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 22:53:28.398891 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 22:53:28.399200 (+ 309us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 22:53:28.399284 (+ 84us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:53:28.400305 (+ 1021us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:53:28.402209 (+ 1904us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:53:28.402950 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:53:28.403031 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 22:53:28.403084 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0929 22:53:28.403218 (+ 134us) server_negotiation.cc:299] Negotiation successful
0929 22:53:28.403260 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":135,"thread_start_us":79,"threads_started":1}
I20260929 22:53:28.403319 31245 tablet_service.cc:1467] Processing CreateTablet for tablet 3658e3215a6947cfb6b38dae0c9f6cdc (DEFAULT_TABLE table=test-table [id=035d5e2be18344649f2c373426f77aab]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 22:53:28.403712 31245 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3658e3215a6947cfb6b38dae0c9f6cdc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 22:53:28.404024 31113 tablet_service.cc:1467] Processing CreateTablet for tablet 3658e3215a6947cfb6b38dae0c9f6cdc (DEFAULT_TABLE table=test-table [id=035d5e2be18344649f2c373426f77aab]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 22:53:28.404275 31113 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3658e3215a6947cfb6b38dae0c9f6cdc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 22:53:28.406301 31321 tablet_bootstrap.cc:492] T 3658e3215a6947cfb6b38dae0c9f6cdc P 4abdea75838e4fbd8bf5d3f863eafea7: Bootstrap starting.
I20260929 22:53:28.406826 31322 tablet_bootstrap.cc:492] T 3658e3215a6947cfb6b38dae0c9f6cdc P 28541c11b79044cf8708e09c266721fc: Bootstrap starting.
I20260929 22:53:28.406972 31321 tablet_bootstrap.cc:654] T 3658e3215a6947cfb6b38dae0c9f6cdc P 4abdea75838e4fbd8bf5d3f863eafea7: Neither blocks nor log segments found. Creating new log.
I20260929 22:53:28.407510 31322 tablet_bootstrap.cc:654] T 3658e3215a6947cfb6b38dae0c9f6cdc P 28541c11b79044cf8708e09c266721fc: Neither blocks nor log segments found. Creating new log.
I20260929 22:53:28.407615 31321 log.cc:824] T 3658e3215a6947cfb6b38dae0c9f6cdc P 4abdea75838e4fbd8bf5d3f863eafea7: Log is configured to *not* fsync() on all Append() calls
I20260929 22:53:28.408021 31322 log.cc:824] T 3658e3215a6947cfb6b38dae0c9f6cdc P 28541c11b79044cf8708e09c266721fc: Log is configured to *not* fsync() on all Append() calls
I20260929 22:53:28.408378 31321 tablet_bootstrap.cc:492] T 3658e3215a6947cfb6b38dae0c9f6cdc P 4abdea75838e4fbd8bf5d3f863eafea7: No bootstrap required, opened a new log
I20260929 22:53:28.408447 31321 ts_tablet_manager.cc:1397] T 3658e3215a6947cfb6b38dae0c9f6cdc P 4abdea75838e4fbd8bf5d3f863eafea7: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260929 22:53:28.408721 31322 tablet_bootstrap.cc:492] T 3658e3215a6947cfb6b38dae0c9f6cdc P 28541c11b79044cf8708e09c266721fc: No bootstrap required, opened a new log
I20260929 22:53:28.408789 31322 ts_tablet_manager.cc:1397] T 3658e3215a6947cfb6b38dae0c9f6cdc P 28541c11b79044cf8708e09c266721fc: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260929 22:53:28.410410 31321 raft_consensus.cc:348] T 3658e3215a6947cfb6b38dae0c9f6cdc P 4abdea75838e4fbd8bf5d3f863eafea7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28541c11b79044cf8708e09c266721fc" member_type: VOTER last_known_addr { host: "127.14.147.194" port: 46873 } } peers { permanent_uuid: "4abdea75838e4fbd8bf5d3f863eafea7" member_type: VOTER last_known_addr { host: "127.14.147.195" port: 40667 } } peers { permanent_uuid: "d1bd8776e4004c71ac8065dc47cf5725" member_type: VOTER last_known_addr { host: "127.14.147.193" port: 42211 } }
I20260929 22:53:28.410624 31321 raft_consensus.cc:374] T 3658e3215a6947cfb6b38dae0c9f6cdc P 4abdea75838e4fbd8bf5d3f863eafea7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 22:53:28.410660 31321 raft_consensus.cc:729] T 3658e3215a6947cfb6b38dae0c9f6cdc P 4abdea75838e4fbd8bf5d3f863eafea7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4abdea75838e4fbd8bf5d3f863eafea7, State: Initialized, Role: FOLLOWER
I20260929 22:53:28.410787 31321 consensus_queue.cc:260] T 3658e3215a6947cfb6b38dae0c9f6cdc P 4abdea75838e4fbd8bf5d3f863eafea7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28541c11b79044cf8708e09c266721fc" member_type: VOTER last_known_addr { host: "127.14.147.194" port: 46873 } } peers { permanent_uuid: "4abdea75838e4fbd8bf5d3f863eafea7" member_type: VOTER last_known_addr { host: "127.14.147.195" port: 40667 } } peers { permanent_uuid: "d1bd8776e4004c71ac8065dc47cf5725" member_type: VOTER last_known_addr { host: "127.14.147.193" port: 42211 } }
I20260929 22:53:28.410806 31322 raft_consensus.cc:348] T 3658e3215a6947cfb6b38dae0c9f6cdc P 28541c11b79044cf8708e09c266721fc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28541c11b79044cf8708e09c266721fc" member_type: VOTER last_known_addr { host: "127.14.147.194" port: 46873 } } peers { permanent_uuid: "4abdea75838e4fbd8bf5d3f863eafea7" member_type: VOTER last_known_addr { host: "127.14.147.195" port: 40667 } } peers { permanent_uuid: "d1bd8776e4004c71ac8065dc47cf5725" member_type: VOTER last_known_addr { host: "127.14.147.193" port: 42211 } }
I20260929 22:53:28.410995 31322 raft_consensus.cc:374] T 3658e3215a6947cfb6b38dae0c9f6cdc P 28541c11b79044cf8708e09c266721fc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 22:53:28.411041 31322 raft_consensus.cc:729] T 3658e3215a6947cfb6b38dae0c9f6cdc P 28541c11b79044cf8708e09c266721fc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 28541c11b79044cf8708e09c266721fc, State: Initialized, Role: FOLLOWER
I20260929 22:53:28.411144 31311 heartbeater.cc:503] Master 127.14.147.254:35531 was elected leader, sending a full tablet report...
I20260929 22:53:28.411149 31322 consensus_queue.cc:260] T 3658e3215a6947cfb6b38dae0c9f6cdc P 28541c11b79044cf8708e09c266721fc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28541c11b79044cf8708e09c266721fc" member_type: VOTER last_known_addr { host: "127.14.147.194" port: 46873 } } peers { permanent_uuid: "4abdea75838e4fbd8bf5d3f863eafea7" member_type: VOTER last_known_addr { host: "127.14.147.195" port: 40667 } } peers { permanent_uuid: "d1bd8776e4004c71ac8065dc47cf5725" member_type: VOTER last_known_addr { host: "127.14.147.193" port: 42211 } }
I20260929 22:53:28.411402 31321 ts_tablet_manager.cc:1428] T 3658e3215a6947cfb6b38dae0c9f6cdc P 4abdea75838e4fbd8bf5d3f863eafea7: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260929 22:53:28.411463 31179 heartbeater.cc:503] Master 127.14.147.254:35531 was elected leader, sending a full tablet report...
I20260929 22:53:28.411451 31322 ts_tablet_manager.cc:1428] T 3658e3215a6947cfb6b38dae0c9f6cdc P 28541c11b79044cf8708e09c266721fc: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
W20260929 22:53:28.501157 31180 tablet.cc:2369] T 3658e3215a6947cfb6b38dae0c9f6cdc P 28541c11b79044cf8708e09c266721fc: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 22:53:28.554724 31047 raft_consensus.cc:482] T 3658e3215a6947cfb6b38dae0c9f6cdc P d1bd8776e4004c71ac8065dc47cf5725 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 22:53:28.554876 31047 raft_consensus.cc:504] T 3658e3215a6947cfb6b38dae0c9f6cdc P d1bd8776e4004c71ac8065dc47cf5725 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28541c11b79044cf8708e09c266721fc" member_type: VOTER last_known_addr { host: "127.14.147.194" port: 46873 } } peers { permanent_uuid: "4abdea75838e4fbd8bf5d3f863eafea7" member_type: VOTER last_known_addr { host: "127.14.147.195" port: 40667 } } peers { permanent_uuid: "d1bd8776e4004c71ac8065dc47cf5725" member_type: VOTER last_known_addr { host: "127.14.147.193" port: 42211 } }
I20260929 22:53:28.555212 31047 leader_election.cc:288] T 3658e3215a6947cfb6b38dae0c9f6cdc P d1bd8776e4004c71ac8065dc47cf5725 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 28541c11b79044cf8708e09c266721fc (127.14.147.194:46873), 4abdea75838e4fbd8bf5d3f863eafea7 (127.14.147.195:40667)
I20260929 22:53:28.558117 31319 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:53:28.555520 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.147.193:41821 (local address 127.14.147.195:40667)
0929 22:53:28.555582 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0929 22:53:28.555584 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 22:53:28.555596 (+ 12us) server_negotiation.cc:407] Connection header received
0929 22:53:28.555635 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 22:53:28.555637 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 22:53:28.555647 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 22:53:28.555704 (+ 57us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 22:53:28.555868 (+ 164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:53:28.556377 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:53:28.557365 (+ 988us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:53:28.557747 (+ 382us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:53:28.557802 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 22:53:28.557809 (+ 7us) server_negotiation.cc:1035] Waiting for connection context
0929 22:53:28.558012 (+ 203us) server_negotiation.cc:299] Negotiation successful
0929 22:53:28.558050 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260929 22:53:28.558123 31043 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:53:28.555396 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.147.195:40667 (local address 127.14.147.193:41821)
0929 22:53:28.555457 (+ 61us) negotiation.cc:107] Waiting for socket to connect
0929 22:53:28.555470 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 22:53:28.555535 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 22:53:28.555721 (+ 186us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 22:53:28.555722 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 22:53:28.555725 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 22:53:28.555801 (+ 76us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 22:53:28.555803 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 22:53:28.556451 (+ 648us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 22:53:28.556451 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 22:53:28.557292 (+ 841us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 22:53:28.557294 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 22:53:28.557818 (+ 524us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 22:53:28.557819 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 22:53:28.557915 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 22:53:28.557944 (+ 29us) client_negotiation.cc:769] Sending connection context
0929 22:53:28.557987 (+ 43us) client_negotiation.cc:240] Negotiation successful
0929 22:53:28.558034 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":39,"mutex_wait_us":24}
I20260929 22:53:28.558128 31318 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:53:28.555660 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.147.193:39997 (local address 127.14.147.194:46873)
0929 22:53:28.555718 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0929 22:53:28.555720 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 22:53:28.555727 (+ 7us) server_negotiation.cc:407] Connection header received
0929 22:53:28.555752 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 22:53:28.555753 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 22:53:28.555760 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 22:53:28.555815 (+ 55us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 22:53:28.555983 (+ 168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:53:28.556447 (+ 464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:53:28.557421 (+ 974us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:53:28.557786 (+ 365us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:53:28.557855 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 22:53:28.557863 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0929 22:53:28.558023 (+ 160us) server_negotiation.cc:299] Negotiation successful
0929 22:53:28.558060 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260929 22:53:28.558147 31327 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:53:28.555398 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.147.194:46873 (local address 127.14.147.193:39997)
0929 22:53:28.555631 (+ 233us) negotiation.cc:107] Waiting for socket to connect
0929 22:53:28.555641 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0929 22:53:28.555691 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 22:53:28.555824 (+ 133us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 22:53:28.555824 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 22:53:28.555826 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 22:53:28.555918 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 22:53:28.555920 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 22:53:28.556522 (+ 602us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 22:53:28.556522 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 22:53:28.557337 (+ 815us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 22:53:28.557340 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 22:53:28.557884 (+ 544us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 22:53:28.557885 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 22:53:28.557936 (+ 51us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 22:53:28.557954 (+ 18us) client_negotiation.cc:769] Sending connection context
0929 22:53:28.558004 (+ 50us) client_negotiation.cc:240] Negotiation successful
0929 22:53:28.558043 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":215,"mutex_wait_us":59,"thread_start_us":104,"threads_started":1}
I20260929 22:53:28.558452 31133 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3658e3215a6947cfb6b38dae0c9f6cdc" candidate_uuid: "d1bd8776e4004c71ac8065dc47cf5725" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "28541c11b79044cf8708e09c266721fc" is_pre_election: true
I20260929 22:53:28.558598 31133 raft_consensus.cc:2454] T 3658e3215a6947cfb6b38dae0c9f6cdc P 28541c11b79044cf8708e09c266721fc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d1bd8776e4004c71ac8065dc47cf5725 in term 0.
I20260929 22:53:28.558660 31265 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3658e3215a6947cfb6b38dae0c9f6cdc" candidate_uuid: "d1bd8776e4004c71ac8065dc47cf5725" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4abdea75838e4fbd8bf5d3f863eafea7" is_pre_election: true
I20260929 22:53:28.558804 31265 raft_consensus.cc:2454] T 3658e3215a6947cfb6b38dae0c9f6cdc P 4abdea75838e4fbd8bf5d3f863eafea7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d1bd8776e4004c71ac8065dc47cf5725 in term 0.
I20260929 22:53:28.558808 30929 leader_election.cc:302] T 3658e3215a6947cfb6b38dae0c9f6cdc P d1bd8776e4004c71ac8065dc47cf5725 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 28541c11b79044cf8708e09c266721fc, d1bd8776e4004c71ac8065dc47cf5725; no voters:
I20260929 22:53:28.558928 31047 raft_consensus.cc:2781] T 3658e3215a6947cfb6b38dae0c9f6cdc P d1bd8776e4004c71ac8065dc47cf5725 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 22:53:28.558998 31047 raft_consensus.cc:482] T 3658e3215a6947cfb6b38dae0c9f6cdc P d1bd8776e4004c71ac8065dc47cf5725 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 22:53:28.559023 31047 raft_consensus.cc:3037] T 3658e3215a6947cfb6b38dae0c9f6cdc P d1bd8776e4004c71ac8065dc47cf5725 [term 0 FOLLOWER]: Advancing to term 1
I20260929 22:53:28.559779 31047 raft_consensus.cc:504] T 3658e3215a6947cfb6b38dae0c9f6cdc P d1bd8776e4004c71ac8065dc47cf5725 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28541c11b79044cf8708e09c266721fc" member_type: VOTER last_known_addr { host: "127.14.147.194" port: 46873 } } peers { permanent_uuid: "4abdea75838e4fbd8bf5d3f863eafea7" member_type: VOTER last_known_addr { host: "127.14.147.195" port: 40667 } } peers { permanent_uuid: "d1bd8776e4004c71ac8065dc47cf5725" member_type: VOTER last_known_addr { host: "127.14.147.193" port: 42211 } }
I20260929 22:53:28.559909 31047 leader_election.cc:288] T 3658e3215a6947cfb6b38dae0c9f6cdc P d1bd8776e4004c71ac8065dc47cf5725 [CANDIDATE]: Term 1 election: Requested vote from peers 28541c11b79044cf8708e09c266721fc (127.14.147.194:46873), 4abdea75838e4fbd8bf5d3f863eafea7 (127.14.147.195:40667)
I20260929 22:53:28.560065 31265 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3658e3215a6947cfb6b38dae0c9f6cdc" candidate_uuid: "d1bd8776e4004c71ac8065dc47cf5725" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4abdea75838e4fbd8bf5d3f863eafea7"
I20260929 22:53:28.560098 31133 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3658e3215a6947cfb6b38dae0c9f6cdc" candidate_uuid: "d1bd8776e4004c71ac8065dc47cf5725" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "28541c11b79044cf8708e09c266721fc"
I20260929 22:53:28.560148 31265 raft_consensus.cc:3037] T 3658e3215a6947cfb6b38dae0c9f6cdc P 4abdea75838e4fbd8bf5d3f863eafea7 [term 0 FOLLOWER]: Advancing to term 1
I20260929 22:53:28.560176 31133 raft_consensus.cc:3037] T 3658e3215a6947cfb6b38dae0c9f6cdc P 28541c11b79044cf8708e09c266721fc [term 0 FOLLOWER]: Advancing to term 1
I20260929 22:53:28.561017 31265 raft_consensus.cc:2454] T 3658e3215a6947cfb6b38dae0c9f6cdc P 4abdea75838e4fbd8bf5d3f863eafea7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d1bd8776e4004c71ac8065dc47cf5725 in term 1.
I20260929 22:53:28.561017 31133 raft_consensus.cc:2454] T 3658e3215a6947cfb6b38dae0c9f6cdc P 28541c11b79044cf8708e09c266721fc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d1bd8776e4004c71ac8065dc47cf5725 in term 1.
I20260929 22:53:28.561197 30928 leader_election.cc:302] T 3658e3215a6947cfb6b38dae0c9f6cdc P d1bd8776e4004c71ac8065dc47cf5725 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4abdea75838e4fbd8bf5d3f863eafea7, d1bd8776e4004c71ac8065dc47cf5725; no voters:
I20260929 22:53:28.561319 31047 raft_consensus.cc:2781] T 3658e3215a6947cfb6b38dae0c9f6cdc P d1bd8776e4004c71ac8065dc47cf5725 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 22:53:28.561470 31047 raft_consensus.cc:686] T 3658e3215a6947cfb6b38dae0c9f6cdc P d1bd8776e4004c71ac8065dc47cf5725 [term 1 LEADER]: Becoming Leader. State: Replica: d1bd8776e4004c71ac8065dc47cf5725, State: Running, Role: LEADER
I20260929 22:53:28.561559 31047 consensus_queue.cc:237] T 3658e3215a6947cfb6b38dae0c9f6cdc P d1bd8776e4004c71ac8065dc47cf5725 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28541c11b79044cf8708e09c266721fc" member_type: VOTER last_known_addr { host: "127.14.147.194" port: 46873 } } peers { permanent_uuid: "4abdea75838e4fbd8bf5d3f863eafea7" member_type: VOTER last_known_addr { host: "127.14.147.195" port: 40667 } } peers { permanent_uuid: "d1bd8776e4004c71ac8065dc47cf5725" member_type: VOTER last_known_addr { host: "127.14.147.193" port: 42211 } }
I20260929 22:53:28.562325 30851 catalog_manager.cc:5495] T 3658e3215a6947cfb6b38dae0c9f6cdc P d1bd8776e4004c71ac8065dc47cf5725 reported cstate change: term changed from 0 to 1, leader changed from <none> to d1bd8776e4004c71ac8065dc47cf5725 (127.14.147.193). New cstate: current_term: 1 leader_uuid: "d1bd8776e4004c71ac8065dc47cf5725" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28541c11b79044cf8708e09c266721fc" member_type: VOTER last_known_addr { host: "127.14.147.194" port: 46873 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4abdea75838e4fbd8bf5d3f863eafea7" member_type: VOTER last_known_addr { host: "127.14.147.195" port: 40667 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d1bd8776e4004c71ac8065dc47cf5725" member_type: VOTER last_known_addr { host: "127.14.147.193" port: 42211 } health_report { overall_health: HEALTHY } } }
I20260929 22:53:28.588775 31049 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:53:28.586448 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38220 (local address 127.14.147.193:42211)
0929 22:53:28.586501 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0929 22:53:28.586503 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 22:53:28.586511 (+ 8us) server_negotiation.cc:407] Connection header received
0929 22:53:28.586555 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 22:53:28.586556 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 22:53:28.586577 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 22:53:28.586646 (+ 69us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 22:53:28.586755 (+ 109us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:53:28.587250 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:53:28.588054 (+ 804us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:53:28.588273 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:53:28.588328 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 22:53:28.588463 (+ 135us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 22:53:28.588565 (+ 102us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 22:53:28.588633 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0929 22:53:28.588669 (+ 36us) server_negotiation.cc:299] Negotiation successful
0929 22:53:28.588694 (+ 25us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260929 22:53:28.590154 31133 raft_consensus.cc:1261] T 3658e3215a6947cfb6b38dae0c9f6cdc P 28541c11b79044cf8708e09c266721fc [term 1 FOLLOWER]: Refusing update from remote peer d1bd8776e4004c71ac8065dc47cf5725: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 22:53:28.590154 31265 raft_consensus.cc:1261] T 3658e3215a6947cfb6b38dae0c9f6cdc P 4abdea75838e4fbd8bf5d3f863eafea7 [term 1 FOLLOWER]: Refusing update from remote peer d1bd8776e4004c71ac8065dc47cf5725: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 22:53:28.590466 31328 consensus_queue.cc:1035] T 3658e3215a6947cfb6b38dae0c9f6cdc P d1bd8776e4004c71ac8065dc47cf5725 [LEADER]: Connected to new peer: Peer: permanent_uuid: "28541c11b79044cf8708e09c266721fc" member_type: VOTER last_known_addr { host: "127.14.147.194" port: 46873 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 22:53:28.590595 31047 consensus_queue.cc:1035] T 3658e3215a6947cfb6b38dae0c9f6cdc P d1bd8776e4004c71ac8065dc47cf5725 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4abdea75838e4fbd8bf5d3f863eafea7" member_type: VOTER last_known_addr { host: "127.14.147.195" port: 40667 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 22:53:28.595193 30852 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49316:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 22:53:28.595274 30852 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49316:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 22:53:28.595964 30852 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 34cef3aa11664e048d242d3feef47476: Sending DeleteTablet for 3 replicas of tablet 3658e3215a6947cfb6b38dae0c9f6cdc
I20260929 22:53:28.596223 30975 tablet_service.cc:1514] Processing DeleteTablet for tablet 3658e3215a6947cfb6b38dae0c9f6cdc with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 22:53:28 UTC) from {username='kudu', principal='kudu/127.14.147.254@KRBTEST.COM'} at 127.0.0.1:38204
I20260929 22:53:28.596290 14927 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhWOCLk/build/release/bin/kudu with pid 30890
I20260929 22:53:28.596309 31050 tablet_replica.cc:330] T 3658e3215a6947cfb6b38dae0c9f6cdc P d1bd8776e4004c71ac8065dc47cf5725: stopping tablet replica
I20260929 22:53:28.596347 31050 raft_consensus.cc:2229] T 3658e3215a6947cfb6b38dae0c9f6cdc P d1bd8776e4004c71ac8065dc47cf5725 [term 1 LEADER]: Raft consensus shutting down.
I20260929 22:53:28.596318 31113 tablet_service.cc:1514] Processing DeleteTablet for tablet 3658e3215a6947cfb6b38dae0c9f6cdc with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 22:53:28 UTC) from {username='kudu', principal='kudu/127.14.147.254@KRBTEST.COM'} at 127.0.0.1:44570
I20260929 22:53:28.596417 31050 raft_consensus.cc:2258] T 3658e3215a6947cfb6b38dae0c9f6cdc P d1bd8776e4004c71ac8065dc47cf5725 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 22:53:28.596515 31245 tablet_service.cc:1514] Processing DeleteTablet for tablet 3658e3215a6947cfb6b38dae0c9f6cdc with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 22:53:28 UTC) from {username='kudu', principal='kudu/127.14.147.254@KRBTEST.COM'} at 127.0.0.1:44394
I20260929 22:53:28.597048 31341 tablet_replica.cc:330] T 3658e3215a6947cfb6b38dae0c9f6cdc P 28541c11b79044cf8708e09c266721fc: stopping tablet replica
I20260929 22:53:28.597148 31341 raft_consensus.cc:2229] T 3658e3215a6947cfb6b38dae0c9f6cdc P 28541c11b79044cf8708e09c266721fc [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 22:53:28.597249 31341 raft_consensus.cc:2258] T 3658e3215a6947cfb6b38dae0c9f6cdc P 28541c11b79044cf8708e09c266721fc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 22:53:28.597644 31341 ts_tablet_manager.cc:1905] T 3658e3215a6947cfb6b38dae0c9f6cdc P 28541c11b79044cf8708e09c266721fc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 22:53:28.599375 31341 ts_tablet_manager.cc:1918] T 3658e3215a6947cfb6b38dae0c9f6cdc P 28541c11b79044cf8708e09c266721fc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 22:53:28.599473 31341 log.cc:1196] T 3658e3215a6947cfb6b38dae0c9f6cdc P 28541c11b79044cf8708e09c266721fc: Deleting WAL directory at /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790722213368738-14927-0/minicluster-data/ts-1/wal/wals/3658e3215a6947cfb6b38dae0c9f6cdc
I20260929 22:53:28.599573 31342 tablet_replica.cc:330] T 3658e3215a6947cfb6b38dae0c9f6cdc P 4abdea75838e4fbd8bf5d3f863eafea7: stopping tablet replica
I20260929 22:53:28.599697 31342 raft_consensus.cc:2229] T 3658e3215a6947cfb6b38dae0c9f6cdc P 4abdea75838e4fbd8bf5d3f863eafea7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 22:53:28.599726 31341 ts_tablet_manager.cc:1939] T 3658e3215a6947cfb6b38dae0c9f6cdc P 28541c11b79044cf8708e09c266721fc: Deleting consensus metadata
I20260929 22:53:28.599808 31342 raft_consensus.cc:2258] T 3658e3215a6947cfb6b38dae0c9f6cdc P 4abdea75838e4fbd8bf5d3f863eafea7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 22:53:28.600136 30839 catalog_manager.cc:4841] TS 28541c11b79044cf8708e09c266721fc (127.14.147.194:46873): tablet 3658e3215a6947cfb6b38dae0c9f6cdc (table test-table [id=035d5e2be18344649f2c373426f77aab]) successfully deleted
I20260929 22:53:28.600215 31342 ts_tablet_manager.cc:1905] T 3658e3215a6947cfb6b38dae0c9f6cdc P 4abdea75838e4fbd8bf5d3f863eafea7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 22:53:28.601583 31342 ts_tablet_manager.cc:1918] T 3658e3215a6947cfb6b38dae0c9f6cdc P 4abdea75838e4fbd8bf5d3f863eafea7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 22:53:28.601647 31342 log.cc:1196] T 3658e3215a6947cfb6b38dae0c9f6cdc P 4abdea75838e4fbd8bf5d3f863eafea7: Deleting WAL directory at /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790722213368738-14927-0/minicluster-data/ts-2/wal/wals/3658e3215a6947cfb6b38dae0c9f6cdc
I20260929 22:53:28.601858 31342 ts_tablet_manager.cc:1939] T 3658e3215a6947cfb6b38dae0c9f6cdc P 4abdea75838e4fbd8bf5d3f863eafea7: Deleting consensus metadata
I20260929 22:53:28.602218 30838 catalog_manager.cc:4841] TS 4abdea75838e4fbd8bf5d3f863eafea7 (127.14.147.195:40667): tablet 3658e3215a6947cfb6b38dae0c9f6cdc (table test-table [id=035d5e2be18344649f2c373426f77aab]) successfully deleted
W20260929 22:53:28.602540 30839 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.14.147.193:42211 (error 108)
I20260929 22:53:28.602797 14927 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhWOCLk/build/release/bin/kudu with pid 31046
I20260929 22:53:28.602980 31048 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:53:28.602869 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.147.193:42211 (local address 127.0.0.1:38236)
0929 22:53:28.602913 (+ 44us) negotiation.cc:107] Waiting for socket to connect
0929 22:53:28.602951 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.147.193:42211: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":27}
W20260929 22:53:28.603408 30839 catalog_manager.cc:4584] TS d1bd8776e4004c71ac8065dc47cf5725 (127.14.147.193:42211): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 3658e3215a6947cfb6b38dae0c9f6cdc: Network error: Client connection negotiation failed: client connection to 127.14.147.193:42211: connect: Connection refused (error 111)
I20260929 22:53:28.607213 14927 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhWOCLk/build/release/bin/kudu with pid 31182
I20260929 22:53:28.611291 14927 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhWOCLk/build/release/bin/kudu with pid 30819
I20260929 22:53:48.616146 14927 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhWOCLk/build/release/bin/kudu
/tmp/dist-test-taskhWOCLk/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790722213368738-14927-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790722213368738-14927-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790722213368738-14927-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790722213368738-14927-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.147.254:35531
--webserver_interface=127.14.147.254
--webserver_port=46623
--keytab_file=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790722213368738-14927-0/krb5kdc/kudu_127.14.147.254.keytab
--principal=kudu/127.14.147.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.147.212:41357
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.147.254:35531
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790722213368738-14927-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790722213368738-14927-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790722213368738-14927-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 22:53:48.699496 31344 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 22:53:48.699723 31344 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 22:53:48.699747 31344 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 22:53:48.700980 31344 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260929 22:53:48.701016 31344 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260929 22:53:48.701028 31344 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260929 22:53:48.701038 31344 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 22:53:48.701049 31344 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 22:53:48.701061 31344 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 22:53:48.701093 31344 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 22:53:48.701104 31344 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 22:53:48.702661 31344 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.147.212:41357
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790722213368738-14927-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790722213368738-14927-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.14.147.254:35531
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790722213368738-14927-0/krb5kdc/kudu_127.14.147.254.keytab
--principal=kudu/127.14.147.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.147.254:35531
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790722213368738-14927-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.14.147.254
--webserver_port=46623
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31344
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790722213368738-14927-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 21:58:54 UTC on ccycloud.quasar-xguoti.root.comops.site
build id 4681390
I20260929 22:53:48.702879 31344 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 22:53:48.703086 31344 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 22:53:48.707891 31349 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 22:53:48.707901 31352 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 22:53:48.707969 31350 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 22:53:48.708189 31344 server_base.cc:1034] running on GCE node
I20260929 22:53:48.708405 31344 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 22:53:48.708669 31344 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 22:53:48.709815 31344 hybrid_clock.cc:648] HybridClock initialized: now 1790722428709799 us; error 30 us; skew 500 ppm
Sep 29 22:53:48 dist-test-slave-98mk krb5kdc[30272](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790722428, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.147.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 22:53:48.712648 31344 init.cc:377] Logged in from keytab as kudu/127.14.147.254@KRBTEST.COM (short username kudu)
I20260929 22:53:48.713145 31344 webserver.cc:460] Webserver started at http://127.14.147.254:46623/ using document root <none> and password file <none>
I20260929 22:53:48.713388 31344 fs_manager.cc:359] Metadata directory not provided
I20260929 22:53:48.713439 31344 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 22:53:48.714871 31344 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260929 22:53:48.715703 31359 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 22:53:48.715967 31344 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260929 22:53:48.716068 31344 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790722213368738-14927-0/minicluster-data/master-0/data,/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790722213368738-14927-0/minicluster-data/master-0/wal
uuid: "34cef3aa11664e048d242d3feef47476"
format_stamp: "Formatted at 2026-09-29 22:53:07 on dist-test-slave-98mk"
server_key: "19382e3e92633e92632c2d8bb0006584"
server_key_iv: "067b6736fb0621bad8f856c90cab1be8"
server_key_version: "encryptionkey@0"
I20260929 22:53:48.716450 31344 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790722213368738-14927-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790722213368738-14927-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790722213368738-14927-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 22:53:48.732581 31344 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 22:53:48.734901 31344 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 22:53:48.735054 31344 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 22:53:48.739094 31344 rpc_server.cc:307] RPC server started. Bound to: 127.14.147.254:35531
I20260929 22:53:48.739228 31411 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.147.254:35531 every 8 connection(s)
I20260929 22:53:48.739543 31344 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790722213368738-14927-0/minicluster-data/master-0/data/info.pb
I20260929 22:53:48.740962 31412 sys_catalog.cc:263] Verifying existing consensus state
I20260929 22:53:48.741407 14927 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhWOCLk/build/release/bin/kudu as pid 31344
I20260929 22:53:48.741650 31412 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 34cef3aa11664e048d242d3feef47476: Bootstrap starting.
I20260929 22:53:48.741662 14927 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhWOCLk/build/release/bin/kudu
/tmp/dist-test-taskhWOCLk/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790722213368738-14927-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790722213368738-14927-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790722213368738-14927-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790722213368738-14927-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.147.193:42211
--local_ip_for_outbound_sockets=127.14.147.193
--tserver_master_addrs=127.14.147.254:35531
--webserver_port=44357
--webserver_interface=127.14.147.193
--keytab_file=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790722213368738-14927-0/krb5kdc/kudu_127.14.147.193.keytab
--principal=kudu/127.14.147.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.147.212:41357
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790722213368738-14927-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790722213368738-14927-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790722213368738-14927-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 22:53:48.749986 31412 log.cc:824] T 00000000000000000000000000000000 P 34cef3aa11664e048d242d3feef47476: Log is configured to *not* fsync() on all Append() calls
I20260929 22:53:48.752264 31412 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 34cef3aa11664e048d242d3feef47476: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=8 ignored=0} mutations{seen=8 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260929 22:53:48.752533 31412 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 34cef3aa11664e048d242d3feef47476: Bootstrap complete.
I20260929 22:53:48.754599 31412 raft_consensus.cc:348] T 00000000000000000000000000000000 P 34cef3aa11664e048d242d3feef47476 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "34cef3aa11664e048d242d3feef47476" member_type: VOTER last_known_addr { host: "127.14.147.254" port: 35531 } }
I20260929 22:53:48.754827 31412 raft_consensus.cc:729] T 00000000000000000000000000000000 P 34cef3aa11664e048d242d3feef47476 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 34cef3aa11664e048d242d3feef47476, State: Initialized, Role: FOLLOWER
I20260929 22:53:48.754976 31412 consensus_queue.cc:260] T 00000000000000000000000000000000 P 34cef3aa11664e048d242d3feef47476 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "34cef3aa11664e048d242d3feef47476" member_type: VOTER last_known_addr { host: "127.14.147.254" port: 35531 } }
I20260929 22:53:48.755050 31412 raft_consensus.cc:388] T 00000000000000000000000000000000 P 34cef3aa11664e048d242d3feef47476 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 22:53:48.755100 31412 raft_consensus.cc:482] T 00000000000000000000000000000000 P 34cef3aa11664e048d242d3feef47476 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 22:53:48.755144 31412 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 34cef3aa11664e048d242d3feef47476 [term 2 FOLLOWER]: Advancing to term 3
I20260929 22:53:48.756187 31412 raft_consensus.cc:504] T 00000000000000000000000000000000 P 34cef3aa11664e048d242d3feef47476 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "34cef3aa11664e048d242d3feef47476" member_type: VOTER last_known_addr { host: "127.14.147.254" port: 35531 } }
I20260929 22:53:48.756284 31412 leader_election.cc:302] T 00000000000000000000000000000000 P 34cef3aa11664e048d242d3feef47476 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 34cef3aa11664e048d242d3feef47476; no voters:
I20260929 22:53:48.756448 31412 leader_election.cc:288] T 00000000000000000000000000000000 P 34cef3aa11664e048d242d3feef47476 [CANDIDATE]: Term 3 election: Requested vote from peers
I20260929 22:53:48.756497 31416 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 34cef3aa11664e048d242d3feef47476 [term 3 FOLLOWER]: Leader election won for term 3
I20260929 22:53:48.756620 31416 raft_consensus.cc:686] T 00000000000000000000000000000000 P 34cef3aa11664e048d242d3feef47476 [term 3 LEADER]: Becoming Leader. State: Replica: 34cef3aa11664e048d242d3feef47476, State: Running, Role: LEADER
I20260929 22:53:48.756716 31416 consensus_queue.cc:237] T 00000000000000000000000000000000 P 34cef3aa11664e048d242d3feef47476 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "34cef3aa11664e048d242d3feef47476" member_type: VOTER last_known_addr { host: "127.14.147.254" port: 35531 } }
I20260929 22:53:48.756719 31412 sys_catalog.cc:564] T 00000000000000000000000000000000 P 34cef3aa11664e048d242d3feef47476 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 22:53:48.757485 31418 sys_catalog.cc:455] T 00000000000000000000000000000000 P 34cef3aa11664e048d242d3feef47476 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 34cef3aa11664e048d242d3feef47476. Latest consensus state: current_term: 3 leader_uuid: "34cef3aa11664e048d242d3feef47476" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "34cef3aa11664e048d242d3feef47476" member_type: VOTER last_known_addr { host: "127.14.147.254" port: 35531 } } }
I20260929 22:53:48.757568 31418 sys_catalog.cc:458] T 00000000000000000000000000000000 P 34cef3aa11664e048d242d3feef47476 [sys.catalog]: This master's current role is: LEADER
I20260929 22:53:48.757702 31417 sys_catalog.cc:455] T 00000000000000000000000000000000 P 34cef3aa11664e048d242d3feef47476 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "34cef3aa11664e048d242d3feef47476" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "34cef3aa11664e048d242d3feef47476" member_type: VOTER last_known_addr { host: "127.14.147.254" port: 35531 } } }
I20260929 22:53:48.757766 31417 sys_catalog.cc:458] T 00000000000000000000000000000000 P 34cef3aa11664e048d242d3feef47476 [sys.catalog]: This master's current role is: LEADER
I20260929 22:53:48.757861 31426 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 22:53:48.758694 31426 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 22:53:48.758800 31426 catalog_manager.cc:1259] Loaded cluster ID: 8c16912f8a7f42e883c53e5491bfb100
I20260929 22:53:48.758828 31426 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 22:53:48.759961 31426 catalog_manager.cc:1504] Loading token signing keys...
I20260929 22:53:48.760156 31426 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 34cef3aa11664e048d242d3feef47476: Loaded TSK: 0
I20260929 22:53:48.760214 31426 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 34cef3aa11664e048d242d3feef47476: Loaded TSK: 1
F20260929 22:53:48.760437 31426 token_signing_key.cc:66] Check failed: _s.ok() Bad status: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7f11cf4b49ad google::LogMessage::Fail() at ??:0
@ 0x7f11cf4b788a google::LogMessage::SendToLog() at ??:0
@ 0x7f11cf4b44de google::LogMessage::Flush() at ??:0
@ 0x7f11cf4b5a19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f11d1c21104 kudu::security::TokenSigningPrivateKey::TokenSigningPrivateKey() at ??:0
@ 0x7f11d1c1ed41 kudu::security::TokenSigner::ImportKeys() at ??:0
@ 0x7f11d740e1cd kudu::master::CatalogManager::LoadTskEntries() at ??:0
@ 0x7f11d740e48e kudu::master::CatalogManager::InitTokenSigner() at ??:0
@ 0x7f11d740e594 _ZNSt17_Function_handlerIFN4kudu6StatusEvEZNS0_6master14CatalogManager24PrepareForLeadershipTaskEvEUlvE3_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x7f11d73d4c48 _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlSt8functionIFNS_6StatusEvEERKNS_9consensus13RaftConsensusElPKcE_clES5_S9_lSB_.isra.591 at ??:0
@ 0x7f11d73dead0 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7f11d13820d3 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f11d1377954 kudu::Thread::SuperviseThread() at ??:0
@ 0x7f11d21db6db start_thread at ??:0
@ 0x7f11cdfb261f clone at ??:0
W20260929 22:53:48.841182 31414 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 22:53:48.841414 31414 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 22:53:48.841439 31414 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 22:53:48.842718 31414 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 22:53:48.842772 31414 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 22:53:48.842811 31414 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.147.193
I20260929 22:53:48.844466 31414 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.147.212:41357
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790722213368738-14927-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790722213368738-14927-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790722213368738-14927-0/krb5kdc/kudu_127.14.147.193.keytab
--principal=kudu/127.14.147.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.147.193:42211
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790722213368738-14927-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.14.147.193
--webserver_port=44357
--tserver_master_addrs=127.14.147.254:35531
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31414
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.147.193
--log_dir=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790722213368738-14927-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 21:58:54 UTC on ccycloud.quasar-xguoti.root.comops.site
build id 4681390
I20260929 22:53:48.844725 31414 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 22:53:48.844938 31414 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 22:53:48.849865 31454 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 22:53:48.849856 31453 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 22:53:48.849862 31456 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 22:53:48.850450 31414 server_base.cc:1034] running on GCE node
I20260929 22:53:48.850642 31414 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 22:53:48.850883 31414 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 22:53:48.852057 31414 hybrid_clock.cc:648] HybridClock initialized: now 1790722428852033 us; error 36 us; skew 500 ppm
Sep 29 22:53:48 dist-test-slave-98mk krb5kdc[30272](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790722428, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.147.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 22:53:48.854545 31414 init.cc:377] Logged in from keytab as kudu/127.14.147.193@KRBTEST.COM (short username kudu)
I20260929 22:53:48.855005 31414 webserver.cc:460] Webserver started at http://127.14.147.193:44357/ using document root <none> and password file <none>
I20260929 22:53:48.855212 31414 fs_manager.cc:359] Metadata directory not provided
I20260929 22:53:48.855261 31414 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 22:53:48.856755 31414 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260929 22:53:48.857452 31463 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 22:53:48.857611 31414 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260929 22:53:48.857694 31414 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790722213368738-14927-0/minicluster-data/ts-0/data,/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790722213368738-14927-0/minicluster-data/ts-0/wal
uuid: "d1bd8776e4004c71ac8065dc47cf5725"
format_stamp: "Formatted at 2026-09-29 22:53:07 on dist-test-slave-98mk"
server_key: "73d4082730f9488ee6a939b600ad086f"
server_key_iv: "bfc24ed012d73abd308ad29d5d8007c3"
server_key_version: "encryptionkey@0"
I20260929 22:53:48.858065 31414 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790722213368738-14927-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790722213368738-14927-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790722213368738-14927-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 22:53:48.867998 31414 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 22:53:48.870230 31414 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 22:53:48.870388 31414 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 22:53:48.870625 31414 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 22:53:48.871088 31470 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260929 22:53:48.873090 31414 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260929 22:53:48.873154 31414 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260929 22:53:48.873193 31414 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260929 22:53:48.873836 31414 ts_tablet_manager.cc:610] Registered 1 tablets
I20260929 22:53:48.873870 31414 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20260929 22:53:48.873961 31470 tablet_bootstrap.cc:492] T 3658e3215a6947cfb6b38dae0c9f6cdc P d1bd8776e4004c71ac8065dc47cf5725: Bootstrap starting.
I20260929 22:53:48.880534 31414 rpc_server.cc:307] RPC server started. Bound to: 127.14.147.193:42211
I20260929 22:53:48.880726 31577 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.147.193:42211 every 8 connection(s)
I20260929 22:53:48.880978 31414 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790722213368738-14927-0/minicluster-data/ts-0/data/info.pb
I20260929 22:53:48.886101 31470 log.cc:824] T 3658e3215a6947cfb6b38dae0c9f6cdc P d1bd8776e4004c71ac8065dc47cf5725: Log is configured to *not* fsync() on all Append() calls
I20260929 22:53:48.886721 14927 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhWOCLk/build/release/bin/kudu as pid 31414
I20260929 22:53:48.886979 14927 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhWOCLk/build/release/bin/kudu
/tmp/dist-test-taskhWOCLk/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790722213368738-14927-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790722213368738-14927-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790722213368738-14927-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790722213368738-14927-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.147.194:46873
--local_ip_for_outbound_sockets=127.14.147.194
--tserver_master_addrs=127.14.147.254:35531
--webserver_port=42367
--webserver_interface=127.14.147.194
--keytab_file=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790722213368738-14927-0/krb5kdc/kudu_127.14.147.194.keytab
--principal=kudu/127.14.147.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.147.212:41357
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790722213368738-14927-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790722213368738-14927-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790722213368738-14927-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 22:53:48.889254 31470 tablet_bootstrap.cc:492] T 3658e3215a6947cfb6b38dae0c9f6cdc P d1bd8776e4004c71ac8065dc47cf5725: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260929 22:53:48.889591 31470 tablet_bootstrap.cc:492] T 3658e3215a6947cfb6b38dae0c9f6cdc P d1bd8776e4004c71ac8065dc47cf5725: Bootstrap complete.
I20260929 22:53:48.890869 31470 ts_tablet_manager.cc:1397] T 3658e3215a6947cfb6b38dae0c9f6cdc P d1bd8776e4004c71ac8065dc47cf5725: Time spent bootstrapping tablet: real 0.017s user 0.008s sys 0.007s
I20260929 22:53:48.892913 31470 raft_consensus.cc:348] T 3658e3215a6947cfb6b38dae0c9f6cdc P d1bd8776e4004c71ac8065dc47cf5725 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28541c11b79044cf8708e09c266721fc" member_type: VOTER last_known_addr { host: "127.14.147.194" port: 46873 } } peers { permanent_uuid: "4abdea75838e4fbd8bf5d3f863eafea7" member_type: VOTER last_known_addr { host: "127.14.147.195" port: 40667 } } peers { permanent_uuid: "d1bd8776e4004c71ac8065dc47cf5725" member_type: VOTER last_known_addr { host: "127.14.147.193" port: 42211 } }
I20260929 22:53:48.893345 31470 raft_consensus.cc:729] T 3658e3215a6947cfb6b38dae0c9f6cdc P d1bd8776e4004c71ac8065dc47cf5725 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: d1bd8776e4004c71ac8065dc47cf5725, State: Initialized, Role: FOLLOWER
I20260929 22:53:48.893548 31470 consensus_queue.cc:260] T 3658e3215a6947cfb6b38dae0c9f6cdc P d1bd8776e4004c71ac8065dc47cf5725 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28541c11b79044cf8708e09c266721fc" member_type: VOTER last_known_addr { host: "127.14.147.194" port: 46873 } } peers { permanent_uuid: "4abdea75838e4fbd8bf5d3f863eafea7" member_type: VOTER last_known_addr { host: "127.14.147.195" port: 40667 } } peers { permanent_uuid: "d1bd8776e4004c71ac8065dc47cf5725" member_type: VOTER last_known_addr { host: "127.14.147.193" port: 42211 } }
I20260929 22:53:48.893875 31470 ts_tablet_manager.cc:1428] T 3658e3215a6947cfb6b38dae0c9f6cdc P d1bd8776e4004c71ac8065dc47cf5725: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
W20260929 22:53:48.961198 31580 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 22:53:48.881667 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.147.254:35531 (local address 127.14.147.193:34367)
0929 22:53:48.881880 (+ 213us) negotiation.cc:107] Waiting for socket to connect
0929 22:53:48.881924 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0929 22:53:48.882395 (+ 471us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 22:53:48.960980 (+ 78585us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.147.254:35531: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":167,"thread_start_us":101,"threads_started":1}
W20260929 22:53:48.961426 31578 heartbeater.cc:644] Failed to heartbeat to 127.14.147.254:35531 (0 consecutive failures): Network error: Failed to ping master at 127.14.147.254:35531: Client connection negotiation failed: client connection to 127.14.147.254:35531: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260929 22:53:48.961798 31580 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:53:48.961659 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.147.254:35531 (local address 127.14.147.193:53179)
0929 22:53:48.961703 (+ 44us) negotiation.cc:107] Waiting for socket to connect
0929 22:53:48.961763 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.147.254:35531: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":18}
I20260929 22:53:48.962107 31580 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:53:48.962034 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.147.254:35531 (local address 127.14.147.193:37279)
0929 22:53:48.962062 (+ 28us) negotiation.cc:107] Waiting for socket to connect
0929 22:53:48.962089 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.147.254:35531: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":14}
W20260929 22:53:48.962205 31578 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260929 22:53:48.972395 31582 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 22:53:48.972605 31582 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 22:53:48.972631 31582 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 22:53:48.973788 31582 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 22:53:48.973824 31582 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 22:53:48.973850 31582 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.147.194
I20260929 22:53:48.975270 31582 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.147.212:41357
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790722213368738-14927-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790722213368738-14927-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790722213368738-14927-0/krb5kdc/kudu_127.14.147.194.keytab
--principal=kudu/127.14.147.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.147.194:46873
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790722213368738-14927-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.14.147.194
--webserver_port=42367
--tserver_master_addrs=127.14.147.254:35531
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31582
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.147.194
--log_dir=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790722213368738-14927-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 21:58:54 UTC on ccycloud.quasar-xguoti.root.comops.site
build id 4681390
I20260929 22:53:48.975508 31582 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 22:53:48.975708 31582 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 22:53:48.980389 31591 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 22:53:48.980387 31589 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 22:53:48.980399 31588 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 22:53:48.980608 31582 server_base.cc:1034] running on GCE node
I20260929 22:53:48.980804 31582 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 22:53:48.981026 31582 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 22:53:48.982198 31582 hybrid_clock.cc:648] HybridClock initialized: now 1790722428982166 us; error 30 us; skew 500 ppm
Sep 29 22:53:48 dist-test-slave-98mk krb5kdc[30272](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790722428, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.147.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 22:53:48.984736 31582 init.cc:377] Logged in from keytab as kudu/127.14.147.194@KRBTEST.COM (short username kudu)
I20260929 22:53:48.985159 31582 webserver.cc:460] Webserver started at http://127.14.147.194:42367/ using document root <none> and password file <none>
I20260929 22:53:48.985369 31582 fs_manager.cc:359] Metadata directory not provided
I20260929 22:53:48.985419 31582 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 22:53:48.986783 31582 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260929 22:53:48.987355 31598 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 22:53:48.987521 31582 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260929 22:53:48.987605 31582 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790722213368738-14927-0/minicluster-data/ts-1/data,/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790722213368738-14927-0/minicluster-data/ts-1/wal
uuid: "28541c11b79044cf8708e09c266721fc"
format_stamp: "Formatted at 2026-09-29 22:53:07 on dist-test-slave-98mk"
server_key: "cfc2dc837f7660f34a1a910a5b5658bf"
server_key_iv: "f2f98582a6acbe96e50692ba98c1f05b"
server_key_version: "encryptionkey@0"
I20260929 22:53:48.987952 31582 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790722213368738-14927-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790722213368738-14927-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790722213368738-14927-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 22:53:49.002022 31582 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 22:53:49.003950 31582 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 22:53:49.004098 31582 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 22:53:49.004335 31582 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 22:53:49.004643 31582 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 22:53:49.004675 31582 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 22:53:49.004704 31582 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 22:53:49.004721 31582 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 22:53:49.010660 31582 rpc_server.cc:307] RPC server started. Bound to: 127.14.147.194:46873
I20260929 22:53:49.010720 31711 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.147.194:46873 every 8 connection(s)
I20260929 22:53:49.011059 31582 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790722213368738-14927-0/minicluster-data/ts-1/data/info.pb
I20260929 22:53:49.012096 31714 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:53:49.011731 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.147.254:35531 (local address 127.14.147.194:49869)
0929 22:53:49.011881 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0929 22:53:49.011955 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.147.254:35531: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":79,"threads_started":1}
W20260929 22:53:49.012486 31712 heartbeater.cc:644] Failed to heartbeat to 127.14.147.254:35531 (0 consecutive failures): Network error: Failed to ping master at 127.14.147.254:35531: Client connection negotiation failed: client connection to 127.14.147.254:35531: connect: Connection refused (error 111)
I20260929 22:53:49.012764 31714 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:53:49.012653 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.147.254:35531 (local address 127.14.147.194:41057)
0929 22:53:49.012701 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0929 22:53:49.012726 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.147.254:35531: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
I20260929 22:53:49.012943 14927 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhWOCLk/build/release/bin/kudu as pid 31582
I20260929 22:53:49.013103 31714 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:53:49.013013 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.147.254:35531 (local address 127.14.147.194:40299)
0929 22:53:49.013064 (+ 51us) negotiation.cc:107] Waiting for socket to connect
0929 22:53:49.013087 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.147.254:35531: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
W20260929 22:53:49.013196 31712 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260929 22:53:49.013211 14927 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhWOCLk/build/release/bin/kudu
/tmp/dist-test-taskhWOCLk/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790722213368738-14927-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790722213368738-14927-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790722213368738-14927-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790722213368738-14927-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.147.195:40667
--local_ip_for_outbound_sockets=127.14.147.195
--tserver_master_addrs=127.14.147.254:35531
--webserver_port=40869
--webserver_interface=127.14.147.195
--keytab_file=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790722213368738-14927-0/krb5kdc/kudu_127.14.147.195.keytab
--principal=kudu/127.14.147.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.147.212:41357
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790722213368738-14927-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790722213368738-14927-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790722213368738-14927-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 22:53:49.103659 31715 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 22:53:49.103878 31715 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 22:53:49.103911 31715 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 22:53:49.105134 31715 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 22:53:49.105177 31715 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 22:53:49.105212 31715 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.147.195
I20260929 22:53:49.106707 31715 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.147.212:41357
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790722213368738-14927-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790722213368738-14927-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790722213368738-14927-0/krb5kdc/kudu_127.14.147.195.keytab
--principal=kudu/127.14.147.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.147.195:40667
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790722213368738-14927-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.14.147.195
--webserver_port=40869
--tserver_master_addrs=127.14.147.254:35531
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31715
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.147.195
--log_dir=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790722213368738-14927-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 21:58:54 UTC on ccycloud.quasar-xguoti.root.comops.site
build id 4681390
I20260929 22:53:49.106951 31715 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 22:53:49.107177 31715 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 22:53:49.112924 31720 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 22:53:49.113269 31715 server_base.cc:1034] running on GCE node
W20260929 22:53:49.115584 31723 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 22:53:49.115991 31721 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 22:53:49.116289 31715 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 22:53:49.116583 31715 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 22:53:49.117717 31715 hybrid_clock.cc:648] HybridClock initialized: now 1790722429117704 us; error 29 us; skew 500 ppm
Sep 29 22:53:49 dist-test-slave-98mk krb5kdc[30272](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790722429, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.147.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 22:53:49.120272 31715 init.cc:377] Logged in from keytab as kudu/127.14.147.195@KRBTEST.COM (short username kudu)
I20260929 22:53:49.120752 31715 webserver.cc:460] Webserver started at http://127.14.147.195:40869/ using document root <none> and password file <none>
I20260929 22:53:49.120985 31715 fs_manager.cc:359] Metadata directory not provided
I20260929 22:53:49.121027 31715 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 22:53:49.122400 31715 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260929 22:53:49.123034 31730 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 22:53:49.123183 31715 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260929 22:53:49.123265 31715 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790722213368738-14927-0/minicluster-data/ts-2/data,/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790722213368738-14927-0/minicluster-data/ts-2/wal
uuid: "4abdea75838e4fbd8bf5d3f863eafea7"
format_stamp: "Formatted at 2026-09-29 22:53:07 on dist-test-slave-98mk"
server_key: "1da1349444d8dccc82d5d454eec06bef"
server_key_iv: "5cdc9391c1f1db9bf81ed2981898a8d6"
server_key_version: "encryptionkey@0"
I20260929 22:53:49.123682 31715 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790722213368738-14927-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790722213368738-14927-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790722213368738-14927-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 22:53:49.130908 31715 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 22:53:49.133050 31715 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 22:53:49.133172 31715 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 22:53:49.133411 31715 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 22:53:49.133710 31715 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 22:53:49.133744 31715 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 22:53:49.133773 31715 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 22:53:49.133791 31715 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 22:53:49.139343 31715 rpc_server.cc:307] RPC server started. Bound to: 127.14.147.195:40667
I20260929 22:53:49.139671 31843 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.147.195:40667 every 8 connection(s)
I20260929 22:53:49.139858 31715 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790722213368738-14927-0/minicluster-data/ts-2/data/info.pb
I20260929 22:53:49.141086 31846 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:53:49.140602 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.147.254:35531 (local address 127.14.147.195:57845)
0929 22:53:49.140779 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0929 22:53:49.140879 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.147.254:35531: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":73,"threads_started":1}
W20260929 22:53:49.141534 31844 heartbeater.cc:644] Failed to heartbeat to 127.14.147.254:35531 (0 consecutive failures): Network error: Failed to ping master at 127.14.147.254:35531: Client connection negotiation failed: client connection to 127.14.147.254:35531: connect: Connection refused (error 111)
I20260929 22:53:49.141880 31846 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:53:49.141762 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.147.254:35531 (local address 127.14.147.195:47375)
0929 22:53:49.141819 (+ 57us) negotiation.cc:107] Waiting for socket to connect
0929 22:53:49.141851 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.147.254:35531: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
I20260929 22:53:49.142195 31846 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:53:49.142107 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.147.254:35531 (local address 127.14.147.195:49271)
0929 22:53:49.142149 (+ 42us) negotiation.cc:107] Waiting for socket to connect
0929 22:53:49.142174 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.147.254:35531: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":26}
W20260929 22:53:49.142315 31844 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260929 22:53:49.148048 14927 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhWOCLk/build/release/bin/kudu as pid 31715
I20260929 22:53:49.962792 31848 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:53:49.962585 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.147.254:35531 (local address 127.14.147.193:35789)
0929 22:53:49.962700 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0929 22:53:49.962745 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.147.254:35531: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":98,"thread_start_us":61,"threads_started":1}
I20260929 22:53:50.013800 31849 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:53:50.013573 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.147.254:35531 (local address 127.14.147.194:37093)
0929 22:53:50.013695 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0929 22:53:50.013736 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.147.254:35531: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":105,"thread_start_us":70,"threads_started":1}
I20260929 22:53:50.142935 31850 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:53:50.142686 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.147.254:35531 (local address 127.14.147.195:53865)
0929 22:53:50.142827 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0929 22:53:50.142873 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.147.254:35531: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":65,"threads_started":1}
I20260929 22:53:50.558768 31851 raft_consensus.cc:482] T 3658e3215a6947cfb6b38dae0c9f6cdc P d1bd8776e4004c71ac8065dc47cf5725 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 22:53:50.558897 31851 raft_consensus.cc:504] T 3658e3215a6947cfb6b38dae0c9f6cdc P d1bd8776e4004c71ac8065dc47cf5725 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28541c11b79044cf8708e09c266721fc" member_type: VOTER last_known_addr { host: "127.14.147.194" port: 46873 } } peers { permanent_uuid: "4abdea75838e4fbd8bf5d3f863eafea7" member_type: VOTER last_known_addr { host: "127.14.147.195" port: 40667 } } peers { permanent_uuid: "d1bd8776e4004c71ac8065dc47cf5725" member_type: VOTER last_known_addr { host: "127.14.147.193" port: 42211 } }
I20260929 22:53:50.559255 31851 leader_election.cc:288] T 3658e3215a6947cfb6b38dae0c9f6cdc P d1bd8776e4004c71ac8065dc47cf5725 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 28541c11b79044cf8708e09c266721fc (127.14.147.194:46873), 4abdea75838e4fbd8bf5d3f863eafea7 (127.14.147.195:40667)
Sep 29 22:53:50 dist-test-slave-98mk krb5kdc[30272](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790722428, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.147.193@KRBTEST.COM for kudu/127.14.147.195@KRBTEST.COM
Sep 29 22:53:50 dist-test-slave-98mk krb5kdc[30272](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790722428, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.147.193@KRBTEST.COM for kudu/127.14.147.194@KRBTEST.COM
I20260929 22:53:50.575520 31852 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:53:50.559473 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.147.194:46873 (local address 127.14.147.193:43439)
0929 22:53:50.559603 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0929 22:53:50.559617 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 22:53:50.559690 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 22:53:50.560560 (+ 870us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 22:53:50.560563 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 22:53:50.560942 (+ 379us) client_negotiation.cc:189] Negotiated authn=SASL
0929 22:53:50.561159 (+ 217us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 22:53:50.561165 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 22:53:50.562319 (+ 1154us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 22:53:50.562320 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 22:53:50.563310 (+ 990us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 22:53:50.563313 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 22:53:50.564375 (+ 1062us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 22:53:50.564376 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 22:53:50.564444 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 22:53:50.569291 (+ 4847us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 22:53:50.569301 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0929 22:53:50.571268 (+ 1967us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 22:53:50.573558 (+ 2290us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 22:53:50.573559 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 22:53:50.573559 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 22:53:50.573691 (+ 132us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 22:53:50.574190 (+ 499us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 22:53:50.574190 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 22:53:50.574191 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 22:53:50.574246 (+ 55us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 22:53:50.574861 (+ 615us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 22:53:50.574871 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 22:53:50.575050 (+ 179us) client_negotiation.cc:769] Sending connection context
0929 22:53:50.575172 (+ 122us) client_negotiation.cc:240] Negotiation successful
0929 22:53:50.575374 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":111,"mutex_wait_us":44,"thread_start_us":46,"threads_started":1}
I20260929 22:53:50.575538 31854 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:53:50.559578 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.147.193:43439 (local address 127.14.147.194:46873)
0929 22:53:50.559744 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0929 22:53:50.559757 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0929 22:53:50.559786 (+ 29us) server_negotiation.cc:407] Connection header received
0929 22:53:50.560326 (+ 540us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 22:53:50.560329 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 22:53:50.560382 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 22:53:50.560510 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0929 22:53:50.561274 (+ 764us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:53:50.562227 (+ 953us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:53:50.563520 (+ 1293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:53:50.564276 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:53:50.564395 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 22:53:50.571397 (+ 7002us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 22:53:50.571401 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 22:53:50.571402 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 22:53:50.571440 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0929 22:53:50.573091 (+ 1651us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 22:53:50.573816 (+ 725us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 22:53:50.573819 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 22:53:50.573819 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 22:53:50.574079 (+ 260us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 22:53:50.574363 (+ 284us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 22:53:50.574365 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 22:53:50.574366 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 22:53:50.574769 (+ 403us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 22:53:50.574867 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0929 22:53:50.575240 (+ 373us) server_negotiation.cc:299] Negotiation successful
0929 22:53:50.575393 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":116,"thread_start_us":61,"threads_started":1}
I20260929 22:53:50.575578 31853 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:53:50.559493 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.147.195:40667 (local address 127.14.147.193:42949)
0929 22:53:50.559736 (+ 243us) negotiation.cc:107] Waiting for socket to connect
0929 22:53:50.559747 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0929 22:53:50.559789 (+ 42us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 22:53:50.560611 (+ 822us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 22:53:50.560612 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 22:53:50.560941 (+ 329us) client_negotiation.cc:189] Negotiated authn=SASL
0929 22:53:50.561159 (+ 218us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 22:53:50.561165 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 22:53:50.562383 (+ 1218us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 22:53:50.562384 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 22:53:50.563203 (+ 819us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 22:53:50.563205 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 22:53:50.563941 (+ 736us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 22:53:50.563942 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 22:53:50.564088 (+ 146us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 22:53:50.569292 (+ 5204us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 22:53:50.569302 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0929 22:53:50.571017 (+ 1715us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 22:53:50.573216 (+ 2199us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 22:53:50.573226 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 22:53:50.573228 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 22:53:50.573475 (+ 247us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 22:53:50.574084 (+ 609us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 22:53:50.574085 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 22:53:50.574085 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 22:53:50.574246 (+ 161us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 22:53:50.574861 (+ 615us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 22:53:50.574871 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 22:53:50.575050 (+ 179us) client_negotiation.cc:769] Sending connection context
0929 22:53:50.575175 (+ 125us) client_negotiation.cc:240] Negotiation successful
0929 22:53:50.575474 (+ 299us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":231,"mutex_wait_us":175,"thread_start_us":52,"threads_started":1}
I20260929 22:53:50.575538 31855 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:53:50.559603 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.147.193:42949 (local address 127.14.147.195:40667)
0929 22:53:50.559758 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0929 22:53:50.559768 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 22:53:50.559876 (+ 108us) server_negotiation.cc:407] Connection header received
0929 22:53:50.560412 (+ 536us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 22:53:50.560415 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 22:53:50.560465 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 22:53:50.560589 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0929 22:53:50.561275 (+ 686us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:53:50.562288 (+ 1013us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:53:50.563270 (+ 982us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:53:50.563788 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:53:50.563870 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 22:53:50.571160 (+ 7290us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 22:53:50.571165 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 22:53:50.571165 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 22:53:50.571178 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0929 22:53:50.573092 (+ 1914us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 22:53:50.573645 (+ 553us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 22:53:50.573652 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 22:53:50.573652 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 22:53:50.573889 (+ 237us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 22:53:50.574365 (+ 476us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 22:53:50.574367 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 22:53:50.574368 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 22:53:50.574770 (+ 402us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 22:53:50.574867 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0929 22:53:50.575240 (+ 373us) server_negotiation.cc:299] Negotiation successful
0929 22:53:50.575392 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":107,"thread_start_us":51,"threads_started":1}
W20260929 22:53:50.576566 31466 leader_election.cc:341] T 3658e3215a6947cfb6b38dae0c9f6cdc P d1bd8776e4004c71ac8065dc47cf5725 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 4abdea75838e4fbd8bf5d3f863eafea7 (127.14.147.195:40667): Not found: Tablet not found: 3658e3215a6947cfb6b38dae0c9f6cdc
W20260929 22:53:50.576644 31467 leader_election.cc:341] T 3658e3215a6947cfb6b38dae0c9f6cdc P d1bd8776e4004c71ac8065dc47cf5725 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 28541c11b79044cf8708e09c266721fc (127.14.147.194:46873): Not found: Tablet not found: 3658e3215a6947cfb6b38dae0c9f6cdc
I20260929 22:53:50.576722 31467 leader_election.cc:302] T 3658e3215a6947cfb6b38dae0c9f6cdc P d1bd8776e4004c71ac8065dc47cf5725 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d1bd8776e4004c71ac8065dc47cf5725; no voters: 28541c11b79044cf8708e09c266721fc, 4abdea75838e4fbd8bf5d3f863eafea7
I20260929 22:53:50.576839 31851 raft_consensus.cc:2726] T 3658e3215a6947cfb6b38dae0c9f6cdc P d1bd8776e4004c71ac8065dc47cf5725 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260929 22:53:50.963510 31852 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:53:50.963337 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.147.254:35531 (local address 127.14.147.193:58879)
0929 22:53:50.963407 (+ 70us) negotiation.cc:107] Waiting for socket to connect
0929 22:53:50.963465 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.147.254:35531: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
I20260929 22:53:51.014518 31856 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:53:51.014287 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.147.254:35531 (local address 127.14.147.194:43673)
0929 22:53:51.014418 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0929 22:53:51.014466 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.147.254:35531: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":75,"threads_started":1}
I20260929 22:53:51.143661 31857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:53:51.143476 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.147.254:35531 (local address 127.14.147.195:38649)
0929 22:53:51.143580 (+ 104us) negotiation.cc:107] Waiting for socket to connect
0929 22:53:51.143614 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.147.254:35531: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":89,"thread_start_us":57,"threads_started":1}
I20260929 22:53:51.964371 31858 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:53:51.964008 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.147.254:35531 (local address 127.14.147.193:56697)
0929 22:53:51.964223 (+ 215us) negotiation.cc:107] Waiting for socket to connect
0929 22:53:51.964300 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.147.254:35531: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":179,"thread_start_us":67,"threads_started":1}
I20260929 22:53:52.015265 31859 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:53:52.015017 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.147.254:35531 (local address 127.14.147.194:50475)
0929 22:53:52.015168 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0929 22:53:52.015213 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.147.254:35531: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":131,"thread_start_us":67,"threads_started":1}
I20260929 22:53:52.144403 31860 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:53:52.144147 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.147.254:35531 (local address 127.14.147.195:43815)
0929 22:53:52.144311 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0929 22:53:52.144356 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.147.254:35531: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":139,"thread_start_us":70,"threads_started":1}
I20260929 22:53:52.783635 31861 raft_consensus.cc:482] T 3658e3215a6947cfb6b38dae0c9f6cdc P d1bd8776e4004c71ac8065dc47cf5725 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 22:53:52.783733 31861 raft_consensus.cc:504] T 3658e3215a6947cfb6b38dae0c9f6cdc P d1bd8776e4004c71ac8065dc47cf5725 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28541c11b79044cf8708e09c266721fc" member_type: VOTER last_known_addr { host: "127.14.147.194" port: 46873 } } peers { permanent_uuid: "4abdea75838e4fbd8bf5d3f863eafea7" member_type: VOTER last_known_addr { host: "127.14.147.195" port: 40667 } } peers { permanent_uuid: "d1bd8776e4004c71ac8065dc47cf5725" member_type: VOTER last_known_addr { host: "127.14.147.193" port: 42211 } }
I20260929 22:53:52.783891 31861 leader_election.cc:288] T 3658e3215a6947cfb6b38dae0c9f6cdc P d1bd8776e4004c71ac8065dc47cf5725 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 28541c11b79044cf8708e09c266721fc (127.14.147.194:46873), 4abdea75838e4fbd8bf5d3f863eafea7 (127.14.147.195:40667)
W20260929 22:53:52.784303 31467 leader_election.cc:341] T 3658e3215a6947cfb6b38dae0c9f6cdc P d1bd8776e4004c71ac8065dc47cf5725 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 28541c11b79044cf8708e09c266721fc (127.14.147.194:46873): Not found: Tablet not found: 3658e3215a6947cfb6b38dae0c9f6cdc
W20260929 22:53:52.784390 31466 leader_election.cc:341] T 3658e3215a6947cfb6b38dae0c9f6cdc P d1bd8776e4004c71ac8065dc47cf5725 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 4abdea75838e4fbd8bf5d3f863eafea7 (127.14.147.195:40667): Not found: Tablet not found: 3658e3215a6947cfb6b38dae0c9f6cdc
I20260929 22:53:52.784430 31466 leader_election.cc:302] T 3658e3215a6947cfb6b38dae0c9f6cdc P d1bd8776e4004c71ac8065dc47cf5725 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d1bd8776e4004c71ac8065dc47cf5725; no voters: 28541c11b79044cf8708e09c266721fc, 4abdea75838e4fbd8bf5d3f863eafea7
I20260929 22:53:52.784504 31861 raft_consensus.cc:2726] T 3658e3215a6947cfb6b38dae0c9f6cdc P d1bd8776e4004c71ac8065dc47cf5725 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260929 22:53:52.965211 31862 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:53:52.964966 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.147.254:35531 (local address 127.14.147.193:39083)
0929 22:53:52.965111 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0929 22:53:52.965158 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.147.254:35531: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":62,"threads_started":1}
I20260929 22:53:53.016062 31863 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:53:53.015854 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.147.254:35531 (local address 127.14.147.194:53471)
0929 22:53:53.015964 (+ 110us) negotiation.cc:107] Waiting for socket to connect
0929 22:53:53.015999 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.147.254:35531: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":95,"thread_start_us":62,"threads_started":1}
I20260929 22:53:53.145315 31864 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:53:53.145025 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.147.254:35531 (local address 127.14.147.195:41839)
0929 22:53:53.145183 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0929 22:53:53.145240 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.147.254:35531: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":136,"thread_start_us":81,"threads_started":1}
I20260929 22:53:53.965951 31865 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:53:53.965683 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.147.254:35531 (local address 127.14.147.193:37875)
0929 22:53:53.965865 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0929 22:53:53.965902 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.147.254:35531: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":164,"thread_start_us":79,"threads_started":1}
I20260929 22:53:54.016829 31866 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:53:54.016576 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.147.254:35531 (local address 127.14.147.194:41819)
0929 22:53:54.016733 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0929 22:53:54.016779 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.147.254:35531: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":131,"thread_start_us":62,"threads_started":1}
I20260929 22:53:54.145994 31867 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:53:54.145769 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.147.254:35531 (local address 127.14.147.195:34843)
0929 22:53:54.145911 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0929 22:53:54.145944 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.147.254:35531: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":88,"threads_started":1}
I20260929 22:53:54.966765 31868 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:53:54.966542 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.147.254:35531 (local address 127.14.147.193:59995)
0929 22:53:54.966669 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0929 22:53:54.966716 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.147.254:35531: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":68,"threads_started":1}
I20260929 22:53:55.017604 31869 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:53:55.017363 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.147.254:35531 (local address 127.14.147.194:50251)
0929 22:53:55.017506 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0929 22:53:55.017554 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.147.254:35531: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":62,"threads_started":1}
I20260929 22:53:55.146716 31870 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:53:55.146454 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.147.254:35531 (local address 127.14.147.195:50353)
0929 22:53:55.146621 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0929 22:53:55.146666 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.147.254:35531: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":144,"thread_start_us":88,"threads_started":1}
I20260929 22:53:55.520314 31871 raft_consensus.cc:482] T 3658e3215a6947cfb6b38dae0c9f6cdc P d1bd8776e4004c71ac8065dc47cf5725 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 22:53:55.520422 31871 raft_consensus.cc:504] T 3658e3215a6947cfb6b38dae0c9f6cdc P d1bd8776e4004c71ac8065dc47cf5725 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28541c11b79044cf8708e09c266721fc" member_type: VOTER last_known_addr { host: "127.14.147.194" port: 46873 } } peers { permanent_uuid: "4abdea75838e4fbd8bf5d3f863eafea7" member_type: VOTER last_known_addr { host: "127.14.147.195" port: 40667 } } peers { permanent_uuid: "d1bd8776e4004c71ac8065dc47cf5725" member_type: VOTER last_known_addr { host: "127.14.147.193" port: 42211 } }
I20260929 22:53:55.520630 31871 leader_election.cc:288] T 3658e3215a6947cfb6b38dae0c9f6cdc P d1bd8776e4004c71ac8065dc47cf5725 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 28541c11b79044cf8708e09c266721fc (127.14.147.194:46873), 4abdea75838e4fbd8bf5d3f863eafea7 (127.14.147.195:40667)
W20260929 22:53:55.520998 31467 leader_election.cc:341] T 3658e3215a6947cfb6b38dae0c9f6cdc P d1bd8776e4004c71ac8065dc47cf5725 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 28541c11b79044cf8708e09c266721fc (127.14.147.194:46873): Not found: Tablet not found: 3658e3215a6947cfb6b38dae0c9f6cdc
W20260929 22:53:55.521077 31466 leader_election.cc:341] T 3658e3215a6947cfb6b38dae0c9f6cdc P d1bd8776e4004c71ac8065dc47cf5725 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 4abdea75838e4fbd8bf5d3f863eafea7 (127.14.147.195:40667): Not found: Tablet not found: 3658e3215a6947cfb6b38dae0c9f6cdc
I20260929 22:53:55.521118 31466 leader_election.cc:302] T 3658e3215a6947cfb6b38dae0c9f6cdc P d1bd8776e4004c71ac8065dc47cf5725 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d1bd8776e4004c71ac8065dc47cf5725; no voters: 28541c11b79044cf8708e09c266721fc, 4abdea75838e4fbd8bf5d3f863eafea7
I20260929 22:53:55.521193 31871 raft_consensus.cc:2726] T 3658e3215a6947cfb6b38dae0c9f6cdc P d1bd8776e4004c71ac8065dc47cf5725 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260929 22:53:55.967624 31872 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:53:55.967300 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.147.254:35531 (local address 127.14.147.193:57575)
0929 22:53:55.967506 (+ 206us) negotiation.cc:107] Waiting for socket to connect
0929 22:53:55.967556 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.147.254:35531: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":183,"thread_start_us":77,"threads_started":1}
I20260929 22:53:56.018221 31873 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:53:56.018008 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.147.254:35531 (local address 127.14.147.194:49303)
0929 22:53:56.018133 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0929 22:53:56.018173 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.147.254:35531: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":75,"threads_started":1}
I20260929 22:53:56.147655 31874 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:53:56.147266 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.147.254:35531 (local address 127.14.147.195:38287)
0929 22:53:56.147541 (+ 275us) negotiation.cc:107] Waiting for socket to connect
0929 22:53:56.147586 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.147.254:35531: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":251,"thread_start_us":85,"threads_started":1}
I20260929 22:53:56.968330 31875 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:53:56.968121 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.147.254:35531 (local address 127.14.147.193:57641)
0929 22:53:56.968241 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0929 22:53:56.968278 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.147.254:35531: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":102,"thread_start_us":62,"threads_started":1}
I20260929 22:53:57.018972 31876 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:53:57.018707 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.147.254:35531 (local address 127.14.147.194:41109)
0929 22:53:57.018873 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0929 22:53:57.018922 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.147.254:35531: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":143,"thread_start_us":82,"threads_started":1}
I20260929 22:53:57.148365 31877 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:53:57.148138 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.147.254:35531 (local address 127.14.147.195:35427)
0929 22:53:57.148279 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0929 22:53:57.148314 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.147.254:35531: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":76,"threads_started":1}
I20260929 22:53:57.458513 31878 raft_consensus.cc:482] T 3658e3215a6947cfb6b38dae0c9f6cdc P d1bd8776e4004c71ac8065dc47cf5725 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 22:53:57.458616 31878 raft_consensus.cc:504] T 3658e3215a6947cfb6b38dae0c9f6cdc P d1bd8776e4004c71ac8065dc47cf5725 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28541c11b79044cf8708e09c266721fc" member_type: VOTER last_known_addr { host: "127.14.147.194" port: 46873 } } peers { permanent_uuid: "4abdea75838e4fbd8bf5d3f863eafea7" member_type: VOTER last_known_addr { host: "127.14.147.195" port: 40667 } } peers { permanent_uuid: "d1bd8776e4004c71ac8065dc47cf5725" member_type: VOTER last_known_addr { host: "127.14.147.193" port: 42211 } }
I20260929 22:53:57.458786 31878 leader_election.cc:288] T 3658e3215a6947cfb6b38dae0c9f6cdc P d1bd8776e4004c71ac8065dc47cf5725 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 28541c11b79044cf8708e09c266721fc (127.14.147.194:46873), 4abdea75838e4fbd8bf5d3f863eafea7 (127.14.147.195:40667)
W20260929 22:53:57.459322 31466 leader_election.cc:341] T 3658e3215a6947cfb6b38dae0c9f6cdc P d1bd8776e4004c71ac8065dc47cf5725 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 4abdea75838e4fbd8bf5d3f863eafea7 (127.14.147.195:40667): Not found: Tablet not found: 3658e3215a6947cfb6b38dae0c9f6cdc
W20260929 22:53:57.459383 31467 leader_election.cc:341] T 3658e3215a6947cfb6b38dae0c9f6cdc P d1bd8776e4004c71ac8065dc47cf5725 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 28541c11b79044cf8708e09c266721fc (127.14.147.194:46873): Not found: Tablet not found: 3658e3215a6947cfb6b38dae0c9f6cdc
I20260929 22:53:57.459474 31467 leader_election.cc:302] T 3658e3215a6947cfb6b38dae0c9f6cdc P d1bd8776e4004c71ac8065dc47cf5725 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d1bd8776e4004c71ac8065dc47cf5725; no voters: 28541c11b79044cf8708e09c266721fc, 4abdea75838e4fbd8bf5d3f863eafea7
I20260929 22:53:57.459550 31878 raft_consensus.cc:2726] T 3658e3215a6947cfb6b38dae0c9f6cdc P d1bd8776e4004c71ac8065dc47cf5725 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260929 22:53:57.969074 31879 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:53:57.968834 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.147.254:35531 (local address 127.14.147.193:40865)
0929 22:53:57.968977 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0929 22:53:57.969025 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.147.254:35531: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":60,"threads_started":1}
I20260929 22:53:58.019799 31880 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:53:58.019522 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.147.254:35531 (local address 127.14.147.194:58779)
0929 22:53:58.019706 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0929 22:53:58.019748 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.147.254:35531: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":167,"thread_start_us":118,"threads_started":1}
I20260929 22:53:58.149073 31881 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:53:58.148827 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.147.254:35531 (local address 127.14.147.195:35789)
0929 22:53:58.148970 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0929 22:53:58.149016 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.147.254:35531: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":62,"threads_started":1}
I20260929 22:53:58.969719 31882 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:53:58.969515 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.147.254:35531 (local address 127.14.147.193:48637)
0929 22:53:58.969629 (+ 114us) negotiation.cc:107] Waiting for socket to connect
0929 22:53:58.969666 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.147.254:35531: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":97,"thread_start_us":63,"threads_started":1}
I20260929 22:53:59.020514 31883 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:53:59.020287 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.147.254:35531 (local address 127.14.147.194:42755)
0929 22:53:59.020410 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0929 22:53:59.020464 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.147.254:35531: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":103,"thread_start_us":63,"threads_started":1}
I20260929 22:53:59.149850 31884 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:53:59.149588 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.147.254:35531 (local address 127.14.147.195:56621)
0929 22:53:59.149743 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0929 22:53:59.149787 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.147.254:35531: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":75,"threads_started":1}
I20260929 22:53:59.970639 31885 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:53:59.970239 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.147.254:35531 (local address 127.14.147.193:55841)
0929 22:53:59.970484 (+ 245us) negotiation.cc:107] Waiting for socket to connect
0929 22:53:59.970567 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.147.254:35531: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":205,"thread_start_us":68,"threads_started":1}
I20260929 22:54:00.021250 31886 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:54:00.021033 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.147.254:35531 (local address 127.14.147.194:56845)
0929 22:54:00.021164 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0929 22:54:00.021201 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.147.254:35531: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":63,"threads_started":1}
I20260929 22:54:00.150701 31887 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:54:00.150461 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.147.254:35531 (local address 127.14.147.195:40785)
0929 22:54:00.150606 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0929 22:54:00.150652 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.147.254:35531: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":64,"threads_started":1}
I20260929 22:54:00.971469 31888 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:54:00.971194 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.147.254:35531 (local address 127.14.147.193:32933)
0929 22:54:00.971347 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0929 22:54:00.971383 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.147.254:35531: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":134,"thread_start_us":66,"threads_started":1}
I20260929 22:54:01.021992 31889 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:54:01.021756 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.147.254:35531 (local address 127.14.147.194:44805)
0929 22:54:01.021903 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0929 22:54:01.021944 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.147.254:35531: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":64,"threads_started":1}
I20260929 22:54:01.151489 31890 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:54:01.151212 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.147.254:35531 (local address 127.14.147.195:42843)
0929 22:54:01.151370 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0929 22:54:01.151411 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.147.254:35531: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":137,"thread_start_us":66,"threads_started":1}
I20260929 22:54:01.561635 31891 raft_consensus.cc:482] T 3658e3215a6947cfb6b38dae0c9f6cdc P d1bd8776e4004c71ac8065dc47cf5725 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 22:54:01.561734 31891 raft_consensus.cc:504] T 3658e3215a6947cfb6b38dae0c9f6cdc P d1bd8776e4004c71ac8065dc47cf5725 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28541c11b79044cf8708e09c266721fc" member_type: VOTER last_known_addr { host: "127.14.147.194" port: 46873 } } peers { permanent_uuid: "4abdea75838e4fbd8bf5d3f863eafea7" member_type: VOTER last_known_addr { host: "127.14.147.195" port: 40667 } } peers { permanent_uuid: "d1bd8776e4004c71ac8065dc47cf5725" member_type: VOTER last_known_addr { host: "127.14.147.193" port: 42211 } }
I20260929 22:54:01.561894 31891 leader_election.cc:288] T 3658e3215a6947cfb6b38dae0c9f6cdc P d1bd8776e4004c71ac8065dc47cf5725 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 28541c11b79044cf8708e09c266721fc (127.14.147.194:46873), 4abdea75838e4fbd8bf5d3f863eafea7 (127.14.147.195:40667)
W20260929 22:54:01.562389 31466 leader_election.cc:341] T 3658e3215a6947cfb6b38dae0c9f6cdc P d1bd8776e4004c71ac8065dc47cf5725 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 4abdea75838e4fbd8bf5d3f863eafea7 (127.14.147.195:40667): Not found: Tablet not found: 3658e3215a6947cfb6b38dae0c9f6cdc
W20260929 22:54:01.562482 31467 leader_election.cc:341] T 3658e3215a6947cfb6b38dae0c9f6cdc P d1bd8776e4004c71ac8065dc47cf5725 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 28541c11b79044cf8708e09c266721fc (127.14.147.194:46873): Not found: Tablet not found: 3658e3215a6947cfb6b38dae0c9f6cdc
I20260929 22:54:01.562530 31467 leader_election.cc:302] T 3658e3215a6947cfb6b38dae0c9f6cdc P d1bd8776e4004c71ac8065dc47cf5725 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d1bd8776e4004c71ac8065dc47cf5725; no voters: 28541c11b79044cf8708e09c266721fc, 4abdea75838e4fbd8bf5d3f863eafea7
I20260929 22:54:01.562637 31891 raft_consensus.cc:2726] T 3658e3215a6947cfb6b38dae0c9f6cdc P d1bd8776e4004c71ac8065dc47cf5725 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260929 22:54:01.972172 31892 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:54:01.971962 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.147.254:35531 (local address 127.14.147.193:36625)
0929 22:54:01.972080 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0929 22:54:01.972119 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.147.254:35531: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":103,"thread_start_us":66,"threads_started":1}
I20260929 22:54:02.022704 31893 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:54:02.022483 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.147.254:35531 (local address 127.14.147.194:58543)
0929 22:54:02.022608 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0929 22:54:02.022655 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.147.254:35531: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":103,"thread_start_us":61,"threads_started":1}
I20260929 22:54:02.152261 31894 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:54:02.152014 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.147.254:35531 (local address 127.14.147.195:46149)
0929 22:54:02.152162 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0929 22:54:02.152209 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.147.254:35531: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":66,"threads_started":1}
I20260929 22:54:02.972985 31895 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:54:02.972728 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.147.254:35531 (local address 127.14.147.193:36041)
0929 22:54:02.972883 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0929 22:54:02.972934 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.147.254:35531: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":64,"threads_started":1}
I20260929 22:54:03.023391 31896 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:54:03.023155 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.147.254:35531 (local address 127.14.147.194:47923)
0929 22:54:03.023295 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0929 22:54:03.023338 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.147.254:35531: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":72,"threads_started":1}
I20260929 22:54:03.153324 31897 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:54:03.152919 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.147.254:35531 (local address 127.14.147.195:40053)
0929 22:54:03.153160 (+ 241us) negotiation.cc:107] Waiting for socket to connect
0929 22:54:03.153241 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.147.254:35531: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":206,"thread_start_us":91,"threads_started":1}
I20260929 22:54:03.973829 31898 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:54:03.973588 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.147.254:35531 (local address 127.14.147.193:42963)
0929 22:54:03.973739 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0929 22:54:03.973779 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.147.254:35531: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":135,"thread_start_us":63,"threads_started":1}
I20260929 22:54:04.024192 31899 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:54:04.023977 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.147.254:35531 (local address 127.14.147.194:52529)
0929 22:54:04.024103 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0929 22:54:04.024146 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.147.254:35531: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":106,"thread_start_us":66,"threads_started":1}
I20260929 22:54:04.152318 14927 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhWOCLk/build/release/bin/kudu with pid 31414
I20260929 22:54:04.154155 31900 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:54:04.153951 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.147.254:35531 (local address 127.14.147.195:49837)
0929 22:54:04.154074 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0929 22:54:04.154107 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.147.254:35531: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":101,"thread_start_us":60,"threads_started":1}
I20260929 22:54:04.156646 14927 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhWOCLk/build/release/bin/kudu with pid 31582
I20260929 22:54:04.160503 14927 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhWOCLk/build/release/bin/kudu with pid 31715
2026-09-29T22:54:04Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedTSK (58142 ms)
[----------] 30 tests from SecurityITest (230798 ms total)
[----------] 6 tests from EncryptionPolicyTest
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/0
Loading random data
Initializing database '/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790722213368738-14927-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 22:54:04 dist-test-slave-98mk krb5kdc[31903](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 22:54:04 dist-test-slave-98mk krb5kdc[31903](info): set up 2 sockets
Sep 29 22:54:04 dist-test-slave-98mk krb5kdc[31903](info): commencing operation
krb5kdc: starting...
W20260929 22:54:05.111557 14927 mini_kdc.cc:121] Time spent starting KDC: real 0.932s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790722213368738-14927-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790722213368738-14927-0/krb5kdc/test-user.keytab.
Sep 29 22:54:05 dist-test-slave-98mk krb5kdc[31903](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790722445, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T22:54:05Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T22:54:05Z Disabled control of system clock
WARNING: no policy specified for kudu/127.14.147.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.147.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.147.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790722213368738-14927-0/krb5kdc/kudu_127.14.147.254.keytab.
Entry for principal kudu/127.14.147.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790722213368738-14927-0/krb5kdc/kudu_127.14.147.254.keytab.
I20260929 22:54:05.232573 14927 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhWOCLk/build/release/bin/kudu
/tmp/dist-test-taskhWOCLk/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790722213368738-14927-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790722213368738-14927-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790722213368738-14927-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790722213368738-14927-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.147.254:46283
--webserver_interface=127.14.147.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790722213368738-14927-0/krb5kdc/kudu_127.14.147.254.keytab
--principal=kudu/127.14.147.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.147.212:38601
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.147.254:46283
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790722213368738-14927-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790722213368738-14927-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790722213368738-14927-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 22:54:05.314332 31916 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 22:54:05.314577 31916 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 22:54:05.314612 31916 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 22:54:05.316318 31916 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 22:54:05.316380 31916 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 22:54:05.316393 31916 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 22:54:05.316406 31916 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 22:54:05.316417 31916 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 22:54:05.318048 31916 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.147.212:38601
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790722213368738-14927-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790722213368738-14927-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.14.147.254:46283
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790722213368738-14927-0/krb5kdc/kudu_127.14.147.254.keytab
--principal=kudu/127.14.147.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.147.254:46283
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790722213368738-14927-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.14.147.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31916
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790722213368738-14927-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 21:58:54 UTC on ccycloud.quasar-xguoti.root.comops.site
build id 4681390
I20260929 22:54:05.318295 31916 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 22:54:05.318531 31916 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 22:54:05.323469 31924 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 22:54:05.323490 31921 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 22:54:05.323463 31922 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 22:54:05.323685 31916 server_base.cc:1034] running on GCE node
I20260929 22:54:05.323915 31916 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 22:54:05.324213 31916 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 22:54:05.325383 31916 hybrid_clock.cc:648] HybridClock initialized: now 1790722445325372 us; error 50 us; skew 500 ppm
Sep 29 22:54:05 dist-test-slave-98mk krb5kdc[31903](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790722445, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.147.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 22:54:05.328045 31916 init.cc:377] Logged in from keytab as kudu/127.14.147.254@KRBTEST.COM (short username kudu)
I20260929 22:54:05.328562 31916 webserver.cc:460] Webserver started at http://127.14.147.254:43471/ using document root <none> and password file <none>
I20260929 22:54:05.328792 31916 fs_manager.cc:359] Metadata directory not provided
I20260929 22:54:05.328845 31916 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 22:54:05.328964 31916 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 22:54:05.330080 31916 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790722213368738-14927-0/minicluster-data/master-0/data/instance:
uuid: "69c0481470814a7898ca4e9b1d52121c"
format_stamp: "Formatted at 2026-09-29 22:54:05 on dist-test-slave-98mk"
server_key: "db69b57903c7ed7ba861dcdca041b0f5"
server_key_iv: "c0ab7a8fa5f24a8a6433b39ade9dab4e"
server_key_version: "encryptionkey@0"
I20260929 22:54:05.330428 31916 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790722213368738-14927-0/minicluster-data/master-0/wal/instance:
uuid: "69c0481470814a7898ca4e9b1d52121c"
format_stamp: "Formatted at 2026-09-29 22:54:05 on dist-test-slave-98mk"
server_key: "db69b57903c7ed7ba861dcdca041b0f5"
server_key_iv: "c0ab7a8fa5f24a8a6433b39ade9dab4e"
server_key_version: "encryptionkey@0"
I20260929 22:54:05.331794 31916 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260929 22:54:05.332701 31931 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 22:54:05.332896 31916 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260929 22:54:05.332978 31916 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790722213368738-14927-0/minicluster-data/master-0/data,/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790722213368738-14927-0/minicluster-data/master-0/wal
uuid: "69c0481470814a7898ca4e9b1d52121c"
format_stamp: "Formatted at 2026-09-29 22:54:05 on dist-test-slave-98mk"
server_key: "db69b57903c7ed7ba861dcdca041b0f5"
server_key_iv: "c0ab7a8fa5f24a8a6433b39ade9dab4e"
server_key_version: "encryptionkey@0"
I20260929 22:54:05.333050 31916 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790722213368738-14927-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790722213368738-14927-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790722213368738-14927-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 22:54:05.347152 31916 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 22:54:05.349331 31916 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 22:54:05.349475 31916 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 22:54:05.353700 31916 rpc_server.cc:307] RPC server started. Bound to: 127.14.147.254:46283
I20260929 22:54:05.353780 31983 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.147.254:46283 every 8 connection(s)
I20260929 22:54:05.354323 31916 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790722213368738-14927-0/minicluster-data/master-0/data/info.pb
I20260929 22:54:05.355271 31984 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 22:54:05.357770 14927 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhWOCLk/build/release/bin/kudu as pid 31916
I20260929 22:54:05.357893 14927 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790722213368738-14927-0/minicluster-data/master-0/wal/instance
I20260929 22:54:05.358052 31984 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 69c0481470814a7898ca4e9b1d52121c: Bootstrap starting.
I20260929 22:54:05.358088 14927 external_mini_cluster.cc:1428] Setting key f1439f5329edc751824bf6f68a6b9adf
I20260929 22:54:05.358748 31984 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 69c0481470814a7898ca4e9b1d52121c: Neither blocks nor log segments found. Creating new log.
I20260929 22:54:05.359026 31984 log.cc:824] T 00000000000000000000000000000000 P 69c0481470814a7898ca4e9b1d52121c: Log is configured to *not* fsync() on all Append() calls
I20260929 22:54:05.359798 31984 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 69c0481470814a7898ca4e9b1d52121c: No bootstrap required, opened a new log
I20260929 22:54:05.361580 31984 raft_consensus.cc:348] T 00000000000000000000000000000000 P 69c0481470814a7898ca4e9b1d52121c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "69c0481470814a7898ca4e9b1d52121c" member_type: VOTER last_known_addr { host: "127.14.147.254" port: 46283 } }
I20260929 22:54:05.361720 31984 raft_consensus.cc:374] T 00000000000000000000000000000000 P 69c0481470814a7898ca4e9b1d52121c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 22:54:05.361750 31984 raft_consensus.cc:729] T 00000000000000000000000000000000 P 69c0481470814a7898ca4e9b1d52121c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 69c0481470814a7898ca4e9b1d52121c, State: Initialized, Role: FOLLOWER
I20260929 22:54:05.361884 31984 consensus_queue.cc:260] T 00000000000000000000000000000000 P 69c0481470814a7898ca4e9b1d52121c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "69c0481470814a7898ca4e9b1d52121c" member_type: VOTER last_known_addr { host: "127.14.147.254" port: 46283 } }
I20260929 22:54:05.361958 31984 raft_consensus.cc:388] T 00000000000000000000000000000000 P 69c0481470814a7898ca4e9b1d52121c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 22:54:05.362000 31984 raft_consensus.cc:482] T 00000000000000000000000000000000 P 69c0481470814a7898ca4e9b1d52121c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 22:54:05.362046 31984 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 69c0481470814a7898ca4e9b1d52121c [term 0 FOLLOWER]: Advancing to term 1
I20260929 22:54:05.362597 31984 raft_consensus.cc:504] T 00000000000000000000000000000000 P 69c0481470814a7898ca4e9b1d52121c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "69c0481470814a7898ca4e9b1d52121c" member_type: VOTER last_known_addr { host: "127.14.147.254" port: 46283 } }
I20260929 22:54:05.362694 31984 leader_election.cc:302] T 00000000000000000000000000000000 P 69c0481470814a7898ca4e9b1d52121c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 69c0481470814a7898ca4e9b1d52121c; no voters:
I20260929 22:54:05.362860 31984 leader_election.cc:288] T 00000000000000000000000000000000 P 69c0481470814a7898ca4e9b1d52121c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 22:54:05.362895 31989 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 69c0481470814a7898ca4e9b1d52121c [term 1 FOLLOWER]: Leader election won for term 1
I20260929 22:54:05.363005 31989 raft_consensus.cc:686] T 00000000000000000000000000000000 P 69c0481470814a7898ca4e9b1d52121c [term 1 LEADER]: Becoming Leader. State: Replica: 69c0481470814a7898ca4e9b1d52121c, State: Running, Role: LEADER
I20260929 22:54:05.363085 31989 consensus_queue.cc:237] T 00000000000000000000000000000000 P 69c0481470814a7898ca4e9b1d52121c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "69c0481470814a7898ca4e9b1d52121c" member_type: VOTER last_known_addr { host: "127.14.147.254" port: 46283 } }
I20260929 22:54:05.363106 31984 sys_catalog.cc:564] T 00000000000000000000000000000000 P 69c0481470814a7898ca4e9b1d52121c [sys.catalog]: configured and running, proceeding with master startup.
I20260929 22:54:05.363441 31991 sys_catalog.cc:455] T 00000000000000000000000000000000 P 69c0481470814a7898ca4e9b1d52121c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "69c0481470814a7898ca4e9b1d52121c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "69c0481470814a7898ca4e9b1d52121c" member_type: VOTER last_known_addr { host: "127.14.147.254" port: 46283 } } }
I20260929 22:54:05.363617 31991 sys_catalog.cc:458] T 00000000000000000000000000000000 P 69c0481470814a7898ca4e9b1d52121c [sys.catalog]: This master's current role is: LEADER
I20260929 22:54:05.363406 31989 sys_catalog.cc:455] T 00000000000000000000000000000000 P 69c0481470814a7898ca4e9b1d52121c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 69c0481470814a7898ca4e9b1d52121c. Latest consensus state: current_term: 1 leader_uuid: "69c0481470814a7898ca4e9b1d52121c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "69c0481470814a7898ca4e9b1d52121c" member_type: VOTER last_known_addr { host: "127.14.147.254" port: 46283 } } }
I20260929 22:54:05.364080 31989 sys_catalog.cc:458] T 00000000000000000000000000000000 P 69c0481470814a7898ca4e9b1d52121c [sys.catalog]: This master's current role is: LEADER
I20260929 22:54:05.364246 31998 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
Sep 29 22:54:05 dist-test-slave-98mk krb5kdc[31903](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790722445, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.147.254@KRBTEST.COM
I20260929 22:54:05.364766 31998 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 22:54:05.366950 31998 catalog_manager.cc:1347] Generated new cluster ID: cb560f9f37ca407fbfb65067ca52135c
I20260929 22:54:05.367002 31998 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 22:54:05.368758 31987 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:54:05.358609 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58530 (local address 127.14.147.254:46283)
0929 22:54:05.358803 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0929 22:54:05.358811 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 22:54:05.358840 (+ 29us) server_negotiation.cc:407] Connection header received
0929 22:54:05.359400 (+ 560us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 22:54:05.359410 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 22:54:05.359517 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 22:54:05.359662 (+ 145us) server_negotiation.cc:226] Negotiated authn=SASL
0929 22:54:05.360308 (+ 646us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:54:05.361204 (+ 896us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:54:05.361990 (+ 786us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:54:05.362454 (+ 464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:54:05.362549 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 22:54:05.365295 (+ 2746us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 22:54:05.365300 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 22:54:05.365300 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 22:54:05.365311 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0929 22:54:05.367138 (+ 1827us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 22:54:05.367563 (+ 425us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 22:54:05.367566 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 22:54:05.367566 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 22:54:05.367671 (+ 105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 22:54:05.367876 (+ 205us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 22:54:05.367877 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 22:54:05.367877 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 22:54:05.368155 (+ 278us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 22:54:05.368303 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0929 22:54:05.368421 (+ 118us) server_negotiation.cc:299] Negotiation successful
0929 22:54:05.368556 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":143,"thread_start_us":74,"threads_started":1}
I20260929 22:54:05.394372 31998 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 22:54:05.394958 31998 catalog_manager.cc:1504] Loading token signing keys...
I20260929 22:54:05.399184 31998 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 69c0481470814a7898ca4e9b1d52121c: Generated new TSK 0
I20260929 22:54:05.399437 31998 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.14.147.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.147.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.147.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790722213368738-14927-0/krb5kdc/kudu_127.14.147.193.keytab.
Entry for principal kudu/127.14.147.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790722213368738-14927-0/krb5kdc/kudu_127.14.147.193.keytab.
I20260929 22:54:05.432145 14927 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhWOCLk/build/release/bin/kudu
/tmp/dist-test-taskhWOCLk/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790722213368738-14927-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790722213368738-14927-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790722213368738-14927-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790722213368738-14927-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.147.193:0
--local_ip_for_outbound_sockets=127.14.147.193
--webserver_interface=127.14.147.193
--webserver_port=0
--tserver_master_addrs=127.14.147.254:46283
--keytab_file=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790722213368738-14927-0/krb5kdc/kudu_127.14.147.193.keytab
--principal=kudu/127.14.147.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.147.212:38601
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790722213368738-14927-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790722213368738-14927-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790722213368738-14927-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 22:54:05.517504 32010 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 22:54:05.517726 32010 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 22:54:05.517756 32010 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 22:54:05.519011 32010 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 22:54:05.519057 32010 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 22:54:05.519085 32010 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.147.193
I20260929 22:54:05.520763 32010 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.147.212:38601
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790722213368738-14927-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790722213368738-14927-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790722213368738-14927-0/krb5kdc/kudu_127.14.147.193.keytab
--principal=kudu/127.14.147.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.147.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790722213368738-14927-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.14.147.193
--webserver_port=0
--tserver_master_addrs=127.14.147.254:46283
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32010
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.147.193
--log_dir=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790722213368738-14927-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 21:58:54 UTC on ccycloud.quasar-xguoti.root.comops.site
build id 4681390
I20260929 22:54:05.521025 32010 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 22:54:05.521245 32010 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 22:54:05.526268 32015 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 22:54:05.526276 32016 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 22:54:05.526268 32018 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 22:54:05.526355 32010 server_base.cc:1034] running on GCE node
I20260929 22:54:05.526688 32010 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 22:54:05.526948 32010 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 22:54:05.528122 32010 hybrid_clock.cc:648] HybridClock initialized: now 1790722445528103 us; error 29 us; skew 500 ppm
Sep 29 22:54:05 dist-test-slave-98mk krb5kdc[31903](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790722445, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.147.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 22:54:05.530735 32010 init.cc:377] Logged in from keytab as kudu/127.14.147.193@KRBTEST.COM (short username kudu)
I20260929 22:54:05.531245 32010 webserver.cc:460] Webserver started at http://127.14.147.193:38411/ using document root <none> and password file <none>
I20260929 22:54:05.531507 32010 fs_manager.cc:359] Metadata directory not provided
I20260929 22:54:05.531559 32010 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 22:54:05.531673 32010 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 22:54:05.532738 32010 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790722213368738-14927-0/minicluster-data/ts-0/data/instance:
uuid: "8614c2a12a604df584db06eca9c6d3b5"
format_stamp: "Formatted at 2026-09-29 22:54:05 on dist-test-slave-98mk"
server_key: "97e0822aed8676c71e8e7c18d560bc8a"
server_key_iv: "dd16ac6cca112811ce389d62be070982"
server_key_version: "encryptionkey@0"
I20260929 22:54:05.533068 32010 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790722213368738-14927-0/minicluster-data/ts-0/wal/instance:
uuid: "8614c2a12a604df584db06eca9c6d3b5"
format_stamp: "Formatted at 2026-09-29 22:54:05 on dist-test-slave-98mk"
server_key: "97e0822aed8676c71e8e7c18d560bc8a"
server_key_iv: "dd16ac6cca112811ce389d62be070982"
server_key_version: "encryptionkey@0"
I20260929 22:54:05.534322 32010 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260929 22:54:05.534981 32025 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 22:54:05.535168 32010 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260929 22:54:05.535271 32010 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790722213368738-14927-0/minicluster-data/ts-0/data,/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790722213368738-14927-0/minicluster-data/ts-0/wal
uuid: "8614c2a12a604df584db06eca9c6d3b5"
format_stamp: "Formatted at 2026-09-29 22:54:05 on dist-test-slave-98mk"
server_key: "97e0822aed8676c71e8e7c18d560bc8a"
server_key_iv: "dd16ac6cca112811ce389d62be070982"
server_key_version: "encryptionkey@0"
I20260929 22:54:05.535332 32010 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790722213368738-14927-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790722213368738-14927-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790722213368738-14927-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 22:54:05.550669 32010 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 22:54:05.552691 32010 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 22:54:05.552809 32010 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 22:54:05.553009 32010 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 22:54:05.553274 32010 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 22:54:05.553306 32010 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 22:54:05.553328 32010 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 22:54:05.553339 32010 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 22:54:05.559727 32010 rpc_server.cc:307] RPC server started. Bound to: 127.14.147.193:34797
I20260929 22:54:05.559803 32139 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.147.193:34797 every 8 connection(s)
I20260929 22:54:05.560127 32010 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790722213368738-14927-0/minicluster-data/ts-0/data/info.pb
Sep 29 22:54:05 dist-test-slave-98mk krb5kdc[31903](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790722445, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.147.193@KRBTEST.COM for kudu/127.14.147.254@KRBTEST.COM
I20260929 22:54:05.567241 14927 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhWOCLk/build/release/bin/kudu as pid 32010
I20260929 22:54:05.567365 14927 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790722213368738-14927-0/minicluster-data/ts-0/wal/instance
I20260929 22:54:05.567567 14927 external_mini_cluster.cc:1428] Setting key bdcaa800c7ac5ced34a45632ff4a96a0
I20260929 22:54:05.569796 31987 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:54:05.561024 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.147.193:56185 (local address 127.14.147.254:46283)
0929 22:54:05.561107 (+ 83us) server_negotiation.cc:206] Beginning negotiation
0929 22:54:05.561110 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 22:54:05.561240 (+ 130us) server_negotiation.cc:407] Connection header received
0929 22:54:05.561737 (+ 497us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 22:54:05.561738 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 22:54:05.561747 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 22:54:05.561792 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0929 22:54:05.562769 (+ 977us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:54:05.563359 (+ 590us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:54:05.564450 (+ 1091us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:54:05.564659 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:54:05.564710 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 22:54:05.566884 (+ 2174us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 22:54:05.566887 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 22:54:05.566888 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 22:54:05.566898 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0929 22:54:05.568437 (+ 1539us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 22:54:05.568813 (+ 376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 22:54:05.568815 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 22:54:05.568815 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 22:54:05.568872 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 22:54:05.569097 (+ 225us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 22:54:05.569098 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 22:54:05.569098 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 22:54:05.569225 (+ 127us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 22:54:05.569292 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0929 22:54:05.569605 (+ 313us) server_negotiation.cc:299] Negotiation successful
0929 22:54:05.569699 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260929 22:54:05.569905 32142 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:54:05.560924 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.147.254:46283 (local address 127.14.147.193:56185)
0929 22:54:05.561107 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0929 22:54:05.561157 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0929 22:54:05.561661 (+ 504us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 22:54:05.561853 (+ 192us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 22:54:05.561856 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 22:54:05.562452 (+ 596us) client_negotiation.cc:189] Negotiated authn=SASL
0929 22:54:05.562656 (+ 204us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 22:54:05.562660 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 22:54:05.563477 (+ 817us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 22:54:05.563478 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 22:54:05.564388 (+ 910us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 22:54:05.564390 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 22:54:05.564749 (+ 359us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 22:54:05.564750 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 22:54:05.564896 (+ 146us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 22:54:05.565371 (+ 475us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 22:54:05.565379 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0929 22:54:05.566729 (+ 1350us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 22:54:05.568516 (+ 1787us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 22:54:05.568524 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 22:54:05.568526 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 22:54:05.568747 (+ 221us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 22:54:05.568935 (+ 188us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 22:54:05.568936 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 22:54:05.568936 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 22:54:05.569043 (+ 107us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 22:54:05.569299 (+ 256us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 22:54:05.569308 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 22:54:05.569481 (+ 173us) client_negotiation.cc:769] Sending connection context
0929 22:54:05.569575 (+ 94us) client_negotiation.cc:240] Negotiation successful
0929 22:54:05.569710 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":69,"threads_started":1}
I20260929 22:54:05.570423 32140 heartbeater.cc:347] Connected to a master server at 127.14.147.254:46283
I20260929 22:54:05.570526 32140 heartbeater.cc:464] Registering TS with master...
I20260929 22:54:05.570719 32140 heartbeater.cc:511] Master 127.14.147.254:46283 requested a full tablet report, sending...
I20260929 22:54:05.571231 31948 ts_manager.cc:194] Registered new tserver with Master: 8614c2a12a604df584db06eca9c6d3b5 (127.14.147.193:34797)
I20260929 22:54:05.571877 31948 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.147.193@KRBTEST.COM'} at 127.14.147.193:56185
WARNING: no policy specified for kudu/127.14.147.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.147.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.147.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790722213368738-14927-0/krb5kdc/kudu_127.14.147.194.keytab.
Entry for principal kudu/127.14.147.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790722213368738-14927-0/krb5kdc/kudu_127.14.147.194.keytab.
I20260929 22:54:05.593976 14927 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhWOCLk/build/release/bin/kudu
/tmp/dist-test-taskhWOCLk/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790722213368738-14927-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790722213368738-14927-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790722213368738-14927-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790722213368738-14927-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.147.194:0
--local_ip_for_outbound_sockets=127.14.147.194
--webserver_interface=127.14.147.194
--webserver_port=0
--tserver_master_addrs=127.14.147.254:46283
--keytab_file=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790722213368738-14927-0/krb5kdc/kudu_127.14.147.194.keytab
--principal=kudu/127.14.147.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.147.212:38601
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790722213368738-14927-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790722213368738-14927-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790722213368738-14927-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 22:54:05.674937 32145 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 22:54:05.675153 32145 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 22:54:05.675179 32145 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 22:54:05.676358 32145 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 22:54:05.676398 32145 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 22:54:05.676424 32145 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.147.194
I20260929 22:54:05.677851 32145 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.147.212:38601
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790722213368738-14927-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790722213368738-14927-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790722213368738-14927-0/krb5kdc/kudu_127.14.147.194.keytab
--principal=kudu/127.14.147.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.147.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790722213368738-14927-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.14.147.194
--webserver_port=0
--tserver_master_addrs=127.14.147.254:46283
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32145
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.147.194
--log_dir=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790722213368738-14927-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 21:58:54 UTC on ccycloud.quasar-xguoti.root.comops.site
build id 4681390
I20260929 22:54:05.678051 32145 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 22:54:05.678244 32145 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 22:54:05.683070 32151 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 22:54:05.683133 32145 server_base.cc:1034] running on GCE node
W20260929 22:54:05.683058 32150 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 22:54:05.683058 32153 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 22:54:05.683502 32145 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 22:54:05.683763 32145 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 22:54:05.684918 32145 hybrid_clock.cc:648] HybridClock initialized: now 1790722445684903 us; error 29 us; skew 500 ppm
Sep 29 22:54:05 dist-test-slave-98mk krb5kdc[31903](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790722445, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.147.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 22:54:05.687244 32145 init.cc:377] Logged in from keytab as kudu/127.14.147.194@KRBTEST.COM (short username kudu)
I20260929 22:54:05.687742 32145 webserver.cc:460] Webserver started at http://127.14.147.194:33925/ using document root <none> and password file <none>
I20260929 22:54:05.687923 32145 fs_manager.cc:359] Metadata directory not provided
I20260929 22:54:05.687961 32145 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 22:54:05.688051 32145 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 22:54:05.689074 32145 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790722213368738-14927-0/minicluster-data/ts-1/data/instance:
uuid: "aba31f38ad0146b8a0239028b93cb1a4"
format_stamp: "Formatted at 2026-09-29 22:54:05 on dist-test-slave-98mk"
server_key: "af53ce187a1842563f1623839abd7118"
server_key_iv: "49c1c50423dc9070954ceddc2b49a4da"
server_key_version: "encryptionkey@0"
I20260929 22:54:05.689360 32145 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790722213368738-14927-0/minicluster-data/ts-1/wal/instance:
uuid: "aba31f38ad0146b8a0239028b93cb1a4"
format_stamp: "Formatted at 2026-09-29 22:54:05 on dist-test-slave-98mk"
server_key: "af53ce187a1842563f1623839abd7118"
server_key_iv: "49c1c50423dc9070954ceddc2b49a4da"
server_key_version: "encryptionkey@0"
I20260929 22:54:05.690557 32145 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260929 22:54:05.691246 32160 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 22:54:05.691473 32145 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260929 22:54:05.691557 32145 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790722213368738-14927-0/minicluster-data/ts-1/data,/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790722213368738-14927-0/minicluster-data/ts-1/wal
uuid: "aba31f38ad0146b8a0239028b93cb1a4"
format_stamp: "Formatted at 2026-09-29 22:54:05 on dist-test-slave-98mk"
server_key: "af53ce187a1842563f1623839abd7118"
server_key_iv: "49c1c50423dc9070954ceddc2b49a4da"
server_key_version: "encryptionkey@0"
I20260929 22:54:05.691618 32145 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790722213368738-14927-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790722213368738-14927-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790722213368738-14927-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 22:54:05.715384 32145 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 22:54:05.717454 32145 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 22:54:05.717568 32145 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 22:54:05.717785 32145 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 22:54:05.718053 32145 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 22:54:05.718083 32145 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 22:54:05.718106 32145 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 22:54:05.718122 32145 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 22:54:05.723694 32145 rpc_server.cc:307] RPC server started. Bound to: 127.14.147.194:45767
I20260929 22:54:05.723733 32273 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.147.194:45767 every 8 connection(s)
I20260929 22:54:05.724043 32145 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790722213368738-14927-0/minicluster-data/ts-1/data/info.pb
I20260929 22:54:05.729753 14927 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhWOCLk/build/release/bin/kudu as pid 32145
I20260929 22:54:05.729856 14927 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790722213368738-14927-0/minicluster-data/ts-1/wal/instance
Sep 29 22:54:05 dist-test-slave-98mk krb5kdc[31903](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790722445, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.147.194@KRBTEST.COM for kudu/127.14.147.254@KRBTEST.COM
I20260929 22:54:05.730182 14927 external_mini_cluster.cc:1428] Setting key 8579e4325032687c153c09a9b0975b32
I20260929 22:54:05.733405 31987 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:54:05.724734 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.147.194:51071 (local address 127.14.147.254:46283)
0929 22:54:05.724815 (+ 81us) server_negotiation.cc:206] Beginning negotiation
0929 22:54:05.724817 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 22:54:05.724975 (+ 158us) server_negotiation.cc:407] Connection header received
0929 22:54:05.725508 (+ 533us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 22:54:05.725509 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 22:54:05.725521 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 22:54:05.725581 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0929 22:54:05.726534 (+ 953us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:54:05.727050 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:54:05.728072 (+ 1022us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:54:05.728271 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:54:05.728326 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 22:54:05.730590 (+ 2264us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 22:54:05.730591 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 22:54:05.730592 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 22:54:05.730600 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0929 22:54:05.731988 (+ 1388us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 22:54:05.732432 (+ 444us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 22:54:05.732433 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 22:54:05.732433 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 22:54:05.732494 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 22:54:05.732747 (+ 253us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 22:54:05.732747 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 22:54:05.732747 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 22:54:05.732849 (+ 102us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 22:54:05.732894 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0929 22:54:05.733246 (+ 352us) server_negotiation.cc:299] Negotiation successful
0929 22:54:05.733331 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260929 22:54:05.733587 32276 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:54:05.724704 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.147.254:46283 (local address 127.14.147.194:51071)
0929 22:54:05.724854 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0929 22:54:05.724892 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0929 22:54:05.725380 (+ 488us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 22:54:05.725655 (+ 275us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 22:54:05.725659 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 22:54:05.726223 (+ 564us) client_negotiation.cc:189] Negotiated authn=SASL
0929 22:54:05.726437 (+ 214us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 22:54:05.726442 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 22:54:05.727117 (+ 675us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 22:54:05.727117 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 22:54:05.727988 (+ 871us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 22:54:05.727991 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 22:54:05.728343 (+ 352us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 22:54:05.728343 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 22:54:05.728443 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 22:54:05.728948 (+ 505us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 22:54:05.728956 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0929 22:54:05.730507 (+ 1551us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 22:54:05.732077 (+ 1570us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 22:54:05.732085 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 22:54:05.732087 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 22:54:05.732355 (+ 268us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 22:54:05.732553 (+ 198us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 22:54:05.732553 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 22:54:05.732554 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 22:54:05.732692 (+ 138us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 22:54:05.732898 (+ 206us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 22:54:05.732903 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 22:54:05.733077 (+ 174us) client_negotiation.cc:769] Sending connection context
0929 22:54:05.733199 (+ 122us) client_negotiation.cc:240] Negotiation successful
0929 22:54:05.733372 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":77,"threads_started":1}
I20260929 22:54:05.734020 32274 heartbeater.cc:347] Connected to a master server at 127.14.147.254:46283
I20260929 22:54:05.734117 32274 heartbeater.cc:464] Registering TS with master...
I20260929 22:54:05.734290 32274 heartbeater.cc:511] Master 127.14.147.254:46283 requested a full tablet report, sending...
I20260929 22:54:05.734712 31948 ts_manager.cc:194] Registered new tserver with Master: aba31f38ad0146b8a0239028b93cb1a4 (127.14.147.194:45767)
I20260929 22:54:05.735220 31948 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.147.194@KRBTEST.COM'} at 127.14.147.194:51071
WARNING: no policy specified for kudu/127.14.147.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.147.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.147.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790722213368738-14927-0/krb5kdc/kudu_127.14.147.195.keytab.
Entry for principal kudu/127.14.147.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790722213368738-14927-0/krb5kdc/kudu_127.14.147.195.keytab.
I20260929 22:54:05.753983 14927 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhWOCLk/build/release/bin/kudu
/tmp/dist-test-taskhWOCLk/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790722213368738-14927-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790722213368738-14927-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790722213368738-14927-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790722213368738-14927-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.147.195:0
--local_ip_for_outbound_sockets=127.14.147.195
--webserver_interface=127.14.147.195
--webserver_port=0
--tserver_master_addrs=127.14.147.254:46283
--keytab_file=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790722213368738-14927-0/krb5kdc/kudu_127.14.147.195.keytab
--principal=kudu/127.14.147.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.147.212:38601
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790722213368738-14927-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790722213368738-14927-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790722213368738-14927-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 22:54:05.834968 32279 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 22:54:05.835175 32279 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 22:54:05.835199 32279 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 22:54:05.836416 32279 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 22:54:05.836460 32279 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 22:54:05.836485 32279 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.147.195
I20260929 22:54:05.837936 32279 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.147.212:38601
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790722213368738-14927-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790722213368738-14927-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790722213368738-14927-0/krb5kdc/kudu_127.14.147.195.keytab
--principal=kudu/127.14.147.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.147.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790722213368738-14927-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.14.147.195
--webserver_port=0
--tserver_master_addrs=127.14.147.254:46283
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32279
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.147.195
--log_dir=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790722213368738-14927-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 21:58:54 UTC on ccycloud.quasar-xguoti.root.comops.site
build id 4681390
I20260929 22:54:05.838132 32279 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 22:54:05.838366 32279 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 22:54:05.843124 32287 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 22:54:05.843146 32284 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 22:54:05.843124 32285 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 22:54:05.843149 32279 server_base.cc:1034] running on GCE node
I20260929 22:54:05.843493 32279 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 22:54:05.843727 32279 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 22:54:05.844916 32279 hybrid_clock.cc:648] HybridClock initialized: now 1790722445844869 us; error 60 us; skew 500 ppm
Sep 29 22:54:05 dist-test-slave-98mk krb5kdc[31903](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790722445, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.147.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 22:54:05.847659 32279 init.cc:377] Logged in from keytab as kudu/127.14.147.195@KRBTEST.COM (short username kudu)
I20260929 22:54:05.848102 32279 webserver.cc:460] Webserver started at http://127.14.147.195:39515/ using document root <none> and password file <none>
I20260929 22:54:05.848328 32279 fs_manager.cc:359] Metadata directory not provided
I20260929 22:54:05.848376 32279 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 22:54:05.848500 32279 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 22:54:05.849560 32279 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790722213368738-14927-0/minicluster-data/ts-2/data/instance:
uuid: "526cb183e1eb40a8b5b5075a2621b2d9"
format_stamp: "Formatted at 2026-09-29 22:54:05 on dist-test-slave-98mk"
server_key: "32f5c4916a43f92b7133ebd7245c9edf"
server_key_iv: "238758b5d0784a3e9e6b01aa7fc3c389"
server_key_version: "encryptionkey@0"
I20260929 22:54:05.849881 32279 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790722213368738-14927-0/minicluster-data/ts-2/wal/instance:
uuid: "526cb183e1eb40a8b5b5075a2621b2d9"
format_stamp: "Formatted at 2026-09-29 22:54:05 on dist-test-slave-98mk"
server_key: "32f5c4916a43f92b7133ebd7245c9edf"
server_key_iv: "238758b5d0784a3e9e6b01aa7fc3c389"
server_key_version: "encryptionkey@0"
I20260929 22:54:05.851265 32279 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.002s
I20260929 22:54:05.851989 32294 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 22:54:05.852130 32279 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260929 22:54:05.852206 32279 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790722213368738-14927-0/minicluster-data/ts-2/data,/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790722213368738-14927-0/minicluster-data/ts-2/wal
uuid: "526cb183e1eb40a8b5b5075a2621b2d9"
format_stamp: "Formatted at 2026-09-29 22:54:05 on dist-test-slave-98mk"
server_key: "32f5c4916a43f92b7133ebd7245c9edf"
server_key_iv: "238758b5d0784a3e9e6b01aa7fc3c389"
server_key_version: "encryptionkey@0"
I20260929 22:54:05.852280 32279 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790722213368738-14927-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790722213368738-14927-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790722213368738-14927-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 22:54:05.868348 32279 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 22:54:05.870407 32279 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 22:54:05.870532 32279 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 22:54:05.870764 32279 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 22:54:05.871052 32279 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 22:54:05.871084 32279 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 22:54:05.871105 32279 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 22:54:05.871129 32279 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 22:54:05.876866 32279 rpc_server.cc:307] RPC server started. Bound to: 127.14.147.195:46053
I20260929 22:54:05.876916 32407 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.147.195:46053 every 8 connection(s)
I20260929 22:54:05.877223 32279 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790722213368738-14927-0/minicluster-data/ts-2/data/info.pb
I20260929 22:54:05.879346 14927 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhWOCLk/build/release/bin/kudu as pid 32279
I20260929 22:54:05.879460 14927 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790722213368738-14927-0/minicluster-data/ts-2/wal/instance
I20260929 22:54:05.879621 14927 external_mini_cluster.cc:1428] Setting key 18dfeebb4069d3015b19c1fd0e76b4f5
Sep 29 22:54:05 dist-test-slave-98mk krb5kdc[31903](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790722445, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.147.195@KRBTEST.COM for kudu/127.14.147.254@KRBTEST.COM
I20260929 22:54:05.886183 31987 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:54:05.877961 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.147.195:56099 (local address 127.14.147.254:46283)
0929 22:54:05.878012 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0929 22:54:05.878015 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 22:54:05.878209 (+ 194us) server_negotiation.cc:407] Connection header received
0929 22:54:05.878681 (+ 472us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 22:54:05.878682 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 22:54:05.878691 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 22:54:05.878744 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0929 22:54:05.879874 (+ 1130us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:54:05.880404 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:54:05.881376 (+ 972us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:54:05.881562 (+ 186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:54:05.881618 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 22:54:05.883824 (+ 2206us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 22:54:05.883826 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 22:54:05.883826 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 22:54:05.883833 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0929 22:54:05.884903 (+ 1070us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 22:54:05.885284 (+ 381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 22:54:05.885285 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 22:54:05.885285 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 22:54:05.885339 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 22:54:05.885563 (+ 224us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 22:54:05.885564 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 22:54:05.885564 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 22:54:05.885678 (+ 114us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 22:54:05.885736 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0929 22:54:05.886010 (+ 274us) server_negotiation.cc:299] Negotiation successful
0929 22:54:05.886093 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260929 22:54:05.886365 32410 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:54:05.877939 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.147.254:46283 (local address 127.14.147.195:56099)
0929 22:54:05.878093 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0929 22:54:05.878129 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0929 22:54:05.878608 (+ 479us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 22:54:05.878797 (+ 189us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 22:54:05.878800 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 22:54:05.879469 (+ 669us) client_negotiation.cc:189] Negotiated authn=SASL
0929 22:54:05.879781 (+ 312us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 22:54:05.879788 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 22:54:05.880495 (+ 707us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 22:54:05.880496 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 22:54:05.881289 (+ 793us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 22:54:05.881292 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 22:54:05.881646 (+ 354us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 22:54:05.881647 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 22:54:05.881768 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 22:54:05.882291 (+ 523us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 22:54:05.882299 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0929 22:54:05.883725 (+ 1426us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 22:54:05.884985 (+ 1260us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 22:54:05.884994 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 22:54:05.884997 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 22:54:05.885216 (+ 219us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 22:54:05.885395 (+ 179us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 22:54:05.885396 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 22:54:05.885396 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 22:54:05.885507 (+ 111us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 22:54:05.885737 (+ 230us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 22:54:05.885744 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 22:54:05.885884 (+ 140us) client_negotiation.cc:769] Sending connection context
0929 22:54:05.885981 (+ 97us) client_negotiation.cc:240] Negotiation successful
0929 22:54:05.886146 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":80,"threads_started":1}
I20260929 22:54:05.886828 32408 heartbeater.cc:347] Connected to a master server at 127.14.147.254:46283
I20260929 22:54:05.886915 32408 heartbeater.cc:464] Registering TS with master...
I20260929 22:54:05.887097 32408 heartbeater.cc:511] Master 127.14.147.254:46283 requested a full tablet report, sending...
I20260929 22:54:05.887498 31948 ts_manager.cc:194] Registered new tserver with Master: 526cb183e1eb40a8b5b5075a2621b2d9 (127.14.147.195:46053)
I20260929 22:54:05.887844 31948 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.147.195@KRBTEST.COM'} at 127.14.147.195:56099
I20260929 22:54:05.891045 14927 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 22:54:05.898226 31987 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:54:05.892306 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58536 (local address 127.14.147.254:46283)
0929 22:54:05.892371 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0929 22:54:05.892373 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 22:54:05.892484 (+ 111us) server_negotiation.cc:407] Connection header received
0929 22:54:05.892507 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 22:54:05.892508 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 22:54:05.892517 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 22:54:05.892557 (+ 40us) server_negotiation.cc:226] Negotiated authn=SASL
0929 22:54:05.893041 (+ 484us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:54:05.893531 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:54:05.894301 (+ 770us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:54:05.894510 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:54:05.894565 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 22:54:05.895724 (+ 1159us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 22:54:05.895725 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 22:54:05.895725 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 22:54:05.895731 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0929 22:54:05.896873 (+ 1142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 22:54:05.897388 (+ 515us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 22:54:05.897389 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 22:54:05.897390 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 22:54:05.897446 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 22:54:05.897732 (+ 286us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 22:54:05.897733 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 22:54:05.897733 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 22:54:05.897902 (+ 169us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 22:54:05.897992 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0929 22:54:05.898086 (+ 94us) server_negotiation.cc:299] Negotiation successful
0929 22:54:05.898153 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260929 22:54:05.899891 31948 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58536:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 22:54:05.901458 31948 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 22:54:05.908159 32424 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:54:05.904088 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37176 (local address 127.14.147.194:45767)
0929 22:54:05.904281 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0929 22:54:05.904285 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 22:54:05.904296 (+ 11us) server_negotiation.cc:407] Connection header received
0929 22:54:05.904336 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 22:54:05.904338 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 22:54:05.904355 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 22:54:05.904651 (+ 296us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 22:54:05.904742 (+ 91us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:54:05.905802 (+ 1060us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:54:05.907027 (+ 1225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:54:05.907769 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:54:05.907847 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 22:54:05.907906 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0929 22:54:05.908051 (+ 145us) server_negotiation.cc:299] Negotiation successful
0929 22:54:05.908088 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":154,"thread_start_us":94,"threads_started":1}
I20260929 22:54:05.908169 32420 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:54:05.903724 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.147.194:45767 (local address 127.0.0.1:37176)
0929 22:54:05.903881 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0929 22:54:05.903922 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0929 22:54:05.903987 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 22:54:05.904488 (+ 501us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 22:54:05.904491 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 22:54:05.904493 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 22:54:05.904574 (+ 81us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 22:54:05.904577 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 22:54:05.905901 (+ 1324us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 22:54:05.905902 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 22:54:05.906953 (+ 1051us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 22:54:05.906954 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 22:54:05.907853 (+ 899us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 22:54:05.907854 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 22:54:05.907938 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 22:54:05.907954 (+ 16us) client_negotiation.cc:769] Sending connection context
0929 22:54:05.908023 (+ 69us) client_negotiation.cc:240] Negotiation successful
0929 22:54:05.908080 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":121,"mutex_wait_us":5,"thread_start_us":44,"threads_started":1}
I20260929 22:54:05.908602 32422 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:54:05.903930 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.147.195:46053 (local address 127.0.0.1:53148)
0929 22:54:05.904365 (+ 435us) negotiation.cc:107] Waiting for socket to connect
0929 22:54:05.904374 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0929 22:54:05.904434 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 22:54:05.904704 (+ 270us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 22:54:05.904705 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 22:54:05.904707 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 22:54:05.904769 (+ 62us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 22:54:05.904770 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 22:54:05.905629 (+ 859us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 22:54:05.905630 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 22:54:05.906904 (+ 1274us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 22:54:05.906906 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 22:54:05.908373 (+ 1467us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 22:54:05.908373 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 22:54:05.908442 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 22:54:05.908446 (+ 4us) client_negotiation.cc:769] Sending connection context
0929 22:54:05.908493 (+ 47us) client_negotiation.cc:240] Negotiation successful
0929 22:54:05.908538 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":412,"thread_start_us":67,"threads_started":1}
I20260929 22:54:05.908900 32423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:54:05.904036 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53148 (local address 127.14.147.195:46053)
0929 22:54:05.904259 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0929 22:54:05.904273 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0929 22:54:05.904490 (+ 217us) server_negotiation.cc:407] Connection header received
0929 22:54:05.904537 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 22:54:05.904539 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 22:54:05.904555 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 22:54:05.904621 (+ 66us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 22:54:05.904835 (+ 214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:54:05.905537 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:54:05.906983 (+ 1446us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:54:05.907770 (+ 787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:54:05.907847 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 22:54:05.907906 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0929 22:54:05.908754 (+ 848us) server_negotiation.cc:299] Negotiation successful
0929 22:54:05.908819 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":130,"thread_start_us":71,"threads_started":1}
I20260929 22:54:05.908927 32419 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:54:05.903735 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.147.193:34797 (local address 127.0.0.1:34998)
0929 22:54:05.904160 (+ 425us) negotiation.cc:107] Waiting for socket to connect
0929 22:54:05.904168 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0929 22:54:05.904237 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 22:54:05.904520 (+ 283us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 22:54:05.904520 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 22:54:05.904522 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 22:54:05.904575 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 22:54:05.904577 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 22:54:05.907036 (+ 2459us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 22:54:05.907037 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 22:54:05.907783 (+ 746us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 22:54:05.907785 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 22:54:05.908723 (+ 938us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 22:54:05.908724 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 22:54:05.908785 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 22:54:05.908788 (+ 3us) client_negotiation.cc:769] Sending connection context
0929 22:54:05.908832 (+ 44us) client_negotiation.cc:240] Negotiation successful
0929 22:54:05.908870 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":412,"mutex_wait_us":3,"thread_start_us":90,"threads_started":1}
I20260929 22:54:05.908921 32208 tablet_service.cc:1467] Processing CreateTablet for tablet be1a55688c10463dac65ea4d0f054e4b (DEFAULT_TABLE table=test-table [id=fdb51a5dd1dc4a059788fc38db013ee3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 22:54:05.909135 32421 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:54:05.903859 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34998 (local address 127.14.147.193:34797)
0929 22:54:05.904028 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0929 22:54:05.904033 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 22:54:05.904294 (+ 261us) server_negotiation.cc:407] Connection header received
0929 22:54:05.904336 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 22:54:05.904338 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 22:54:05.904355 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 22:54:05.904444 (+ 89us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 22:54:05.904650 (+ 206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:54:05.905800 (+ 1150us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:54:05.907872 (+ 2072us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:54:05.908574 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:54:05.908633 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 22:54:05.908688 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0929 22:54:05.909014 (+ 326us) server_negotiation.cc:299] Negotiation successful
0929 22:54:05.909065 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":137,"thread_start_us":66,"threads_started":1}
I20260929 22:54:05.909322 32208 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet be1a55688c10463dac65ea4d0f054e4b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 22:54:05.909786 32073 tablet_service.cc:1467] Processing CreateTablet for tablet be1a55688c10463dac65ea4d0f054e4b (DEFAULT_TABLE table=test-table [id=fdb51a5dd1dc4a059788fc38db013ee3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 22:54:05.910051 32073 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet be1a55688c10463dac65ea4d0f054e4b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 22:54:05.912093 32425 tablet_bootstrap.cc:492] T be1a55688c10463dac65ea4d0f054e4b P aba31f38ad0146b8a0239028b93cb1a4: Bootstrap starting.
I20260929 22:54:05.912125 32342 tablet_service.cc:1467] Processing CreateTablet for tablet be1a55688c10463dac65ea4d0f054e4b (DEFAULT_TABLE table=test-table [id=fdb51a5dd1dc4a059788fc38db013ee3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 22:54:05.912441 32342 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet be1a55688c10463dac65ea4d0f054e4b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 22:54:05.912766 32426 tablet_bootstrap.cc:492] T be1a55688c10463dac65ea4d0f054e4b P 8614c2a12a604df584db06eca9c6d3b5: Bootstrap starting.
I20260929 22:54:05.912792 32425 tablet_bootstrap.cc:654] T be1a55688c10463dac65ea4d0f054e4b P aba31f38ad0146b8a0239028b93cb1a4: Neither blocks nor log segments found. Creating new log.
I20260929 22:54:05.913213 32425 log.cc:824] T be1a55688c10463dac65ea4d0f054e4b P aba31f38ad0146b8a0239028b93cb1a4: Log is configured to *not* fsync() on all Append() calls
I20260929 22:54:05.913372 32426 tablet_bootstrap.cc:654] T be1a55688c10463dac65ea4d0f054e4b P 8614c2a12a604df584db06eca9c6d3b5: Neither blocks nor log segments found. Creating new log.
I20260929 22:54:05.913816 32425 tablet_bootstrap.cc:492] T be1a55688c10463dac65ea4d0f054e4b P aba31f38ad0146b8a0239028b93cb1a4: No bootstrap required, opened a new log
I20260929 22:54:05.913861 32425 ts_tablet_manager.cc:1397] T be1a55688c10463dac65ea4d0f054e4b P aba31f38ad0146b8a0239028b93cb1a4: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260929 22:54:05.913918 32426 log.cc:824] T be1a55688c10463dac65ea4d0f054e4b P 8614c2a12a604df584db06eca9c6d3b5: Log is configured to *not* fsync() on all Append() calls
I20260929 22:54:05.914660 32426 tablet_bootstrap.cc:492] T be1a55688c10463dac65ea4d0f054e4b P 8614c2a12a604df584db06eca9c6d3b5: No bootstrap required, opened a new log
I20260929 22:54:05.914717 32426 ts_tablet_manager.cc:1397] T be1a55688c10463dac65ea4d0f054e4b P 8614c2a12a604df584db06eca9c6d3b5: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260929 22:54:05.915282 32425 raft_consensus.cc:348] T be1a55688c10463dac65ea4d0f054e4b P aba31f38ad0146b8a0239028b93cb1a4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aba31f38ad0146b8a0239028b93cb1a4" member_type: VOTER last_known_addr { host: "127.14.147.194" port: 45767 } } peers { permanent_uuid: "8614c2a12a604df584db06eca9c6d3b5" member_type: VOTER last_known_addr { host: "127.14.147.193" port: 34797 } } peers { permanent_uuid: "526cb183e1eb40a8b5b5075a2621b2d9" member_type: VOTER last_known_addr { host: "127.14.147.195" port: 46053 } }
I20260929 22:54:05.915452 32425 raft_consensus.cc:374] T be1a55688c10463dac65ea4d0f054e4b P aba31f38ad0146b8a0239028b93cb1a4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 22:54:05.915489 32425 raft_consensus.cc:729] T be1a55688c10463dac65ea4d0f054e4b P aba31f38ad0146b8a0239028b93cb1a4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aba31f38ad0146b8a0239028b93cb1a4, State: Initialized, Role: FOLLOWER
I20260929 22:54:05.915571 32429 tablet_bootstrap.cc:492] T be1a55688c10463dac65ea4d0f054e4b P 526cb183e1eb40a8b5b5075a2621b2d9: Bootstrap starting.
I20260929 22:54:05.915647 32425 consensus_queue.cc:260] T be1a55688c10463dac65ea4d0f054e4b P aba31f38ad0146b8a0239028b93cb1a4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aba31f38ad0146b8a0239028b93cb1a4" member_type: VOTER last_known_addr { host: "127.14.147.194" port: 45767 } } peers { permanent_uuid: "8614c2a12a604df584db06eca9c6d3b5" member_type: VOTER last_known_addr { host: "127.14.147.193" port: 34797 } } peers { permanent_uuid: "526cb183e1eb40a8b5b5075a2621b2d9" member_type: VOTER last_known_addr { host: "127.14.147.195" port: 46053 } }
I20260929 22:54:05.915935 32425 ts_tablet_manager.cc:1428] T be1a55688c10463dac65ea4d0f054e4b P aba31f38ad0146b8a0239028b93cb1a4: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260929 22:54:05.916054 32274 heartbeater.cc:503] Master 127.14.147.254:46283 was elected leader, sending a full tablet report...
I20260929 22:54:05.916203 32429 tablet_bootstrap.cc:654] T be1a55688c10463dac65ea4d0f054e4b P 526cb183e1eb40a8b5b5075a2621b2d9: Neither blocks nor log segments found. Creating new log.
I20260929 22:54:05.916728 32429 log.cc:824] T be1a55688c10463dac65ea4d0f054e4b P 526cb183e1eb40a8b5b5075a2621b2d9: Log is configured to *not* fsync() on all Append() calls
I20260929 22:54:05.916657 32426 raft_consensus.cc:348] T be1a55688c10463dac65ea4d0f054e4b P 8614c2a12a604df584db06eca9c6d3b5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aba31f38ad0146b8a0239028b93cb1a4" member_type: VOTER last_known_addr { host: "127.14.147.194" port: 45767 } } peers { permanent_uuid: "8614c2a12a604df584db06eca9c6d3b5" member_type: VOTER last_known_addr { host: "127.14.147.193" port: 34797 } } peers { permanent_uuid: "526cb183e1eb40a8b5b5075a2621b2d9" member_type: VOTER last_known_addr { host: "127.14.147.195" port: 46053 } }
I20260929 22:54:05.916826 32426 raft_consensus.cc:374] T be1a55688c10463dac65ea4d0f054e4b P 8614c2a12a604df584db06eca9c6d3b5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 22:54:05.916853 32426 raft_consensus.cc:729] T be1a55688c10463dac65ea4d0f054e4b P 8614c2a12a604df584db06eca9c6d3b5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8614c2a12a604df584db06eca9c6d3b5, State: Initialized, Role: FOLLOWER
I20260929 22:54:05.916939 32426 consensus_queue.cc:260] T be1a55688c10463dac65ea4d0f054e4b P 8614c2a12a604df584db06eca9c6d3b5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aba31f38ad0146b8a0239028b93cb1a4" member_type: VOTER last_known_addr { host: "127.14.147.194" port: 45767 } } peers { permanent_uuid: "8614c2a12a604df584db06eca9c6d3b5" member_type: VOTER last_known_addr { host: "127.14.147.193" port: 34797 } } peers { permanent_uuid: "526cb183e1eb40a8b5b5075a2621b2d9" member_type: VOTER last_known_addr { host: "127.14.147.195" port: 46053 } }
I20260929 22:54:05.917229 32426 ts_tablet_manager.cc:1428] T be1a55688c10463dac65ea4d0f054e4b P 8614c2a12a604df584db06eca9c6d3b5: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260929 22:54:05.917249 32140 heartbeater.cc:503] Master 127.14.147.254:46283 was elected leader, sending a full tablet report...
I20260929 22:54:05.917493 32429 tablet_bootstrap.cc:492] T be1a55688c10463dac65ea4d0f054e4b P 526cb183e1eb40a8b5b5075a2621b2d9: No bootstrap required, opened a new log
I20260929 22:54:05.917562 32429 ts_tablet_manager.cc:1397] T be1a55688c10463dac65ea4d0f054e4b P 526cb183e1eb40a8b5b5075a2621b2d9: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260929 22:54:05.919034 32429 raft_consensus.cc:348] T be1a55688c10463dac65ea4d0f054e4b P 526cb183e1eb40a8b5b5075a2621b2d9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aba31f38ad0146b8a0239028b93cb1a4" member_type: VOTER last_known_addr { host: "127.14.147.194" port: 45767 } } peers { permanent_uuid: "8614c2a12a604df584db06eca9c6d3b5" member_type: VOTER last_known_addr { host: "127.14.147.193" port: 34797 } } peers { permanent_uuid: "526cb183e1eb40a8b5b5075a2621b2d9" member_type: VOTER last_known_addr { host: "127.14.147.195" port: 46053 } }
I20260929 22:54:05.919216 32429 raft_consensus.cc:374] T be1a55688c10463dac65ea4d0f054e4b P 526cb183e1eb40a8b5b5075a2621b2d9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 22:54:05.919256 32429 raft_consensus.cc:729] T be1a55688c10463dac65ea4d0f054e4b P 526cb183e1eb40a8b5b5075a2621b2d9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 526cb183e1eb40a8b5b5075a2621b2d9, State: Initialized, Role: FOLLOWER
I20260929 22:54:05.919363 32429 consensus_queue.cc:260] T be1a55688c10463dac65ea4d0f054e4b P 526cb183e1eb40a8b5b5075a2621b2d9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aba31f38ad0146b8a0239028b93cb1a4" member_type: VOTER last_known_addr { host: "127.14.147.194" port: 45767 } } peers { permanent_uuid: "8614c2a12a604df584db06eca9c6d3b5" member_type: VOTER last_known_addr { host: "127.14.147.193" port: 34797 } } peers { permanent_uuid: "526cb183e1eb40a8b5b5075a2621b2d9" member_type: VOTER last_known_addr { host: "127.14.147.195" port: 46053 } }
I20260929 22:54:05.919632 32429 ts_tablet_manager.cc:1428] T be1a55688c10463dac65ea4d0f054e4b P 526cb183e1eb40a8b5b5075a2621b2d9: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260929 22:54:05.919663 32408 heartbeater.cc:503] Master 127.14.147.254:46283 was elected leader, sending a full tablet report...
I20260929 22:54:05.919871 32430 raft_consensus.cc:482] T be1a55688c10463dac65ea4d0f054e4b P aba31f38ad0146b8a0239028b93cb1a4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 22:54:05.919971 32430 raft_consensus.cc:504] T be1a55688c10463dac65ea4d0f054e4b P aba31f38ad0146b8a0239028b93cb1a4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aba31f38ad0146b8a0239028b93cb1a4" member_type: VOTER last_known_addr { host: "127.14.147.194" port: 45767 } } peers { permanent_uuid: "8614c2a12a604df584db06eca9c6d3b5" member_type: VOTER last_known_addr { host: "127.14.147.193" port: 34797 } } peers { permanent_uuid: "526cb183e1eb40a8b5b5075a2621b2d9" member_type: VOTER last_known_addr { host: "127.14.147.195" port: 46053 } }
I20260929 22:54:05.920303 32430 leader_election.cc:288] T be1a55688c10463dac65ea4d0f054e4b P aba31f38ad0146b8a0239028b93cb1a4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8614c2a12a604df584db06eca9c6d3b5 (127.14.147.193:34797), 526cb183e1eb40a8b5b5075a2621b2d9 (127.14.147.195:46053)
I20260929 22:54:05.923544 32421 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:54:05.920521 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.147.194:51469 (local address 127.14.147.193:34797)
0929 22:54:05.920697 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0929 22:54:05.920699 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 22:54:05.920706 (+ 7us) server_negotiation.cc:407] Connection header received
0929 22:54:05.920724 (+ 18us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 22:54:05.920725 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 22:54:05.920732 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 22:54:05.920776 (+ 44us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 22:54:05.920933 (+ 157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:54:05.921475 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:54:05.922480 (+ 1005us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:54:05.922938 (+ 458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:54:05.922995 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 22:54:05.923003 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0929 22:54:05.923405 (+ 402us) server_negotiation.cc:299] Negotiation successful
0929 22:54:05.923484 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":154}
I20260929 22:54:05.923686 32276 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:54:05.920458 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.147.193:34797 (local address 127.14.147.194:51469)
0929 22:54:05.920494 (+ 36us) negotiation.cc:107] Waiting for socket to connect
0929 22:54:05.920506 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0929 22:54:05.920546 (+ 40us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 22:54:05.920795 (+ 249us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 22:54:05.920796 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 22:54:05.920798 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 22:54:05.920875 (+ 77us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 22:54:05.920877 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 22:54:05.921549 (+ 672us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 22:54:05.921550 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 22:54:05.922392 (+ 842us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 22:54:05.922395 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 22:54:05.923097 (+ 702us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 22:54:05.923099 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 22:54:05.923257 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 22:54:05.923266 (+ 9us) client_negotiation.cc:769] Sending connection context
0929 22:54:05.923390 (+ 124us) client_negotiation.cc:240] Negotiation successful
0929 22:54:05.923605 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":15}
I20260929 22:54:05.923965 32434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:54:05.920552 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.147.195:46053 (local address 127.14.147.194:58455)
0929 22:54:05.920693 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0929 22:54:05.920709 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 22:54:05.920753 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 22:54:05.921061 (+ 308us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 22:54:05.921062 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 22:54:05.921064 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 22:54:05.921135 (+ 71us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 22:54:05.921137 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 22:54:05.921814 (+ 677us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 22:54:05.921815 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 22:54:05.922593 (+ 778us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 22:54:05.922596 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 22:54:05.923657 (+ 1061us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 22:54:05.923658 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 22:54:05.923754 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 22:54:05.923759 (+ 5us) client_negotiation.cc:769] Sending connection context
0929 22:54:05.923846 (+ 87us) client_negotiation.cc:240] Negotiation successful
0929 22:54:05.923890 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":91,"threads_started":1}
I20260929 22:54:05.924005 32423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:54:05.920872 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.147.194:58455 (local address 127.14.147.195:46053)
0929 22:54:05.920963 (+ 91us) server_negotiation.cc:206] Beginning negotiation
0929 22:54:05.920965 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 22:54:05.920972 (+ 7us) server_negotiation.cc:407] Connection header received
0929 22:54:05.920992 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 22:54:05.920992 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 22:54:05.921000 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 22:54:05.921043 (+ 43us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 22:54:05.921185 (+ 142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:54:05.921732 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:54:05.922759 (+ 1027us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:54:05.923350 (+ 591us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:54:05.923554 (+ 204us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 22:54:05.923571 (+ 17us) server_negotiation.cc:1035] Waiting for connection context
0929 22:54:05.923866 (+ 295us) server_negotiation.cc:299] Negotiation successful
0929 22:54:05.923943 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260929 22:54:05.924033 32093 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "be1a55688c10463dac65ea4d0f054e4b" candidate_uuid: "aba31f38ad0146b8a0239028b93cb1a4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8614c2a12a604df584db06eca9c6d3b5" is_pre_election: true
I20260929 22:54:05.924178 32093 raft_consensus.cc:2454] T be1a55688c10463dac65ea4d0f054e4b P 8614c2a12a604df584db06eca9c6d3b5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate aba31f38ad0146b8a0239028b93cb1a4 in term 0.
I20260929 22:54:05.924307 32362 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "be1a55688c10463dac65ea4d0f054e4b" candidate_uuid: "aba31f38ad0146b8a0239028b93cb1a4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "526cb183e1eb40a8b5b5075a2621b2d9" is_pre_election: true
I20260929 22:54:05.924389 32164 leader_election.cc:302] T be1a55688c10463dac65ea4d0f054e4b P aba31f38ad0146b8a0239028b93cb1a4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8614c2a12a604df584db06eca9c6d3b5, aba31f38ad0146b8a0239028b93cb1a4; no voters:
I20260929 22:54:05.924440 32362 raft_consensus.cc:2454] T be1a55688c10463dac65ea4d0f054e4b P 526cb183e1eb40a8b5b5075a2621b2d9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate aba31f38ad0146b8a0239028b93cb1a4 in term 0.
I20260929 22:54:05.924512 32430 raft_consensus.cc:2781] T be1a55688c10463dac65ea4d0f054e4b P aba31f38ad0146b8a0239028b93cb1a4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 22:54:05.924584 32430 raft_consensus.cc:482] T be1a55688c10463dac65ea4d0f054e4b P aba31f38ad0146b8a0239028b93cb1a4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 22:54:05.924614 32430 raft_consensus.cc:3037] T be1a55688c10463dac65ea4d0f054e4b P aba31f38ad0146b8a0239028b93cb1a4 [term 0 FOLLOWER]: Advancing to term 1
I20260929 22:54:05.925312 32430 raft_consensus.cc:504] T be1a55688c10463dac65ea4d0f054e4b P aba31f38ad0146b8a0239028b93cb1a4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aba31f38ad0146b8a0239028b93cb1a4" member_type: VOTER last_known_addr { host: "127.14.147.194" port: 45767 } } peers { permanent_uuid: "8614c2a12a604df584db06eca9c6d3b5" member_type: VOTER last_known_addr { host: "127.14.147.193" port: 34797 } } peers { permanent_uuid: "526cb183e1eb40a8b5b5075a2621b2d9" member_type: VOTER last_known_addr { host: "127.14.147.195" port: 46053 } }
I20260929 22:54:05.925431 32430 leader_election.cc:288] T be1a55688c10463dac65ea4d0f054e4b P aba31f38ad0146b8a0239028b93cb1a4 [CANDIDATE]: Term 1 election: Requested vote from peers 8614c2a12a604df584db06eca9c6d3b5 (127.14.147.193:34797), 526cb183e1eb40a8b5b5075a2621b2d9 (127.14.147.195:46053)
I20260929 22:54:05.925614 32362 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "be1a55688c10463dac65ea4d0f054e4b" candidate_uuid: "aba31f38ad0146b8a0239028b93cb1a4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "526cb183e1eb40a8b5b5075a2621b2d9"
I20260929 22:54:05.925616 32093 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "be1a55688c10463dac65ea4d0f054e4b" candidate_uuid: "aba31f38ad0146b8a0239028b93cb1a4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8614c2a12a604df584db06eca9c6d3b5"
I20260929 22:54:05.925683 32093 raft_consensus.cc:3037] T be1a55688c10463dac65ea4d0f054e4b P 8614c2a12a604df584db06eca9c6d3b5 [term 0 FOLLOWER]: Advancing to term 1
I20260929 22:54:05.925683 32362 raft_consensus.cc:3037] T be1a55688c10463dac65ea4d0f054e4b P 526cb183e1eb40a8b5b5075a2621b2d9 [term 0 FOLLOWER]: Advancing to term 1
I20260929 22:54:05.926321 32362 raft_consensus.cc:2454] T be1a55688c10463dac65ea4d0f054e4b P 526cb183e1eb40a8b5b5075a2621b2d9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate aba31f38ad0146b8a0239028b93cb1a4 in term 1.
I20260929 22:54:05.926375 32093 raft_consensus.cc:2454] T be1a55688c10463dac65ea4d0f054e4b P 8614c2a12a604df584db06eca9c6d3b5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate aba31f38ad0146b8a0239028b93cb1a4 in term 1.
I20260929 22:54:05.926450 32164 leader_election.cc:302] T be1a55688c10463dac65ea4d0f054e4b P aba31f38ad0146b8a0239028b93cb1a4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 526cb183e1eb40a8b5b5075a2621b2d9, aba31f38ad0146b8a0239028b93cb1a4; no voters:
I20260929 22:54:05.926554 32430 raft_consensus.cc:2781] T be1a55688c10463dac65ea4d0f054e4b P aba31f38ad0146b8a0239028b93cb1a4 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 22:54:05.926695 32430 raft_consensus.cc:686] T be1a55688c10463dac65ea4d0f054e4b P aba31f38ad0146b8a0239028b93cb1a4 [term 1 LEADER]: Becoming Leader. State: Replica: aba31f38ad0146b8a0239028b93cb1a4, State: Running, Role: LEADER
I20260929 22:54:05.926776 32430 consensus_queue.cc:237] T be1a55688c10463dac65ea4d0f054e4b P aba31f38ad0146b8a0239028b93cb1a4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aba31f38ad0146b8a0239028b93cb1a4" member_type: VOTER last_known_addr { host: "127.14.147.194" port: 45767 } } peers { permanent_uuid: "8614c2a12a604df584db06eca9c6d3b5" member_type: VOTER last_known_addr { host: "127.14.147.193" port: 34797 } } peers { permanent_uuid: "526cb183e1eb40a8b5b5075a2621b2d9" member_type: VOTER last_known_addr { host: "127.14.147.195" port: 46053 } }
I20260929 22:54:05.927588 31948 catalog_manager.cc:5495] T be1a55688c10463dac65ea4d0f054e4b P aba31f38ad0146b8a0239028b93cb1a4 reported cstate change: term changed from 0 to 1, leader changed from <none> to aba31f38ad0146b8a0239028b93cb1a4 (127.14.147.194). New cstate: current_term: 1 leader_uuid: "aba31f38ad0146b8a0239028b93cb1a4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aba31f38ad0146b8a0239028b93cb1a4" member_type: VOTER last_known_addr { host: "127.14.147.194" port: 45767 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8614c2a12a604df584db06eca9c6d3b5" member_type: VOTER last_known_addr { host: "127.14.147.193" port: 34797 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "526cb183e1eb40a8b5b5075a2621b2d9" member_type: VOTER last_known_addr { host: "127.14.147.195" port: 46053 } health_report { overall_health: UNKNOWN } } }
I20260929 22:54:05.932931 32424 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:54:05.930645 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37184 (local address 127.14.147.194:45767)
0929 22:54:05.930687 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0929 22:54:05.930689 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 22:54:05.930698 (+ 9us) server_negotiation.cc:407] Connection header received
0929 22:54:05.930770 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 22:54:05.930771 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 22:54:05.930789 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 22:54:05.930850 (+ 61us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 22:54:05.930954 (+ 104us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:54:05.931503 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:54:05.932235 (+ 732us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:54:05.932441 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:54:05.932493 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 22:54:05.932630 (+ 137us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 22:54:05.932734 (+ 104us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 22:54:05.932803 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0929 22:54:05.932842 (+ 39us) server_negotiation.cc:299] Negotiation successful
0929 22:54:05.932862 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260929 22:54:05.934649 32362 raft_consensus.cc:1261] T be1a55688c10463dac65ea4d0f054e4b P 526cb183e1eb40a8b5b5075a2621b2d9 [term 1 FOLLOWER]: Refusing update from remote peer aba31f38ad0146b8a0239028b93cb1a4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 22:54:05.934657 32093 raft_consensus.cc:1261] T be1a55688c10463dac65ea4d0f054e4b P 8614c2a12a604df584db06eca9c6d3b5 [term 1 FOLLOWER]: Refusing update from remote peer aba31f38ad0146b8a0239028b93cb1a4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 22:54:05.934947 32430 consensus_queue.cc:1035] T be1a55688c10463dac65ea4d0f054e4b P aba31f38ad0146b8a0239028b93cb1a4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "526cb183e1eb40a8b5b5075a2621b2d9" member_type: VOTER last_known_addr { host: "127.14.147.195" port: 46053 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 22:54:05.935086 32435 consensus_queue.cc:1035] T be1a55688c10463dac65ea4d0f054e4b P aba31f38ad0146b8a0239028b93cb1a4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8614c2a12a604df584db06eca9c6d3b5" member_type: VOTER last_known_addr { host: "127.14.147.193" port: 34797 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 22:54:05.937775 32438 mvcc.cc:205] Tried to move back new op lower bound from 7334799138545893376 to 7334799138516721664. Current Snapshot: MvccSnapshot[applied={T|T < 7334799138545893376}]
I20260929 22:54:05.939692 31948 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58536:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 22:54:05.939780 31948 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58536:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 22:54:05.940474 31948 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 69c0481470814a7898ca4e9b1d52121c: Sending DeleteTablet for 3 replicas of tablet be1a55688c10463dac65ea4d0f054e4b
I20260929 22:54:05.940757 32208 tablet_service.cc:1514] Processing DeleteTablet for tablet be1a55688c10463dac65ea4d0f054e4b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 22:54:05 UTC) from {username='kudu', principal='kudu/127.14.147.254@KRBTEST.COM'} at 127.0.0.1:37176
I20260929 22:54:05.940812 32342 tablet_service.cc:1514] Processing DeleteTablet for tablet be1a55688c10463dac65ea4d0f054e4b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 22:54:05 UTC) from {username='kudu', principal='kudu/127.14.147.254@KRBTEST.COM'} at 127.0.0.1:53148
I20260929 22:54:05.940830 32073 tablet_service.cc:1514] Processing DeleteTablet for tablet be1a55688c10463dac65ea4d0f054e4b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 22:54:05 UTC) from {username='kudu', principal='kudu/127.14.147.254@KRBTEST.COM'} at 127.0.0.1:34998
I20260929 22:54:05.941018 32450 tablet_replica.cc:330] T be1a55688c10463dac65ea4d0f054e4b P 8614c2a12a604df584db06eca9c6d3b5: stopping tablet replica
I20260929 22:54:05.941114 32450 raft_consensus.cc:2229] T be1a55688c10463dac65ea4d0f054e4b P 8614c2a12a604df584db06eca9c6d3b5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 22:54:05.941170 32449 tablet_replica.cc:330] T be1a55688c10463dac65ea4d0f054e4b P 526cb183e1eb40a8b5b5075a2621b2d9: stopping tablet replica
I20260929 22:54:05.941216 32450 raft_consensus.cc:2258] T be1a55688c10463dac65ea4d0f054e4b P 8614c2a12a604df584db06eca9c6d3b5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 22:54:05.941250 32449 raft_consensus.cc:2229] T be1a55688c10463dac65ea4d0f054e4b P 526cb183e1eb40a8b5b5075a2621b2d9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 22:54:05.941327 32449 raft_consensus.cc:2258] T be1a55688c10463dac65ea4d0f054e4b P 526cb183e1eb40a8b5b5075a2621b2d9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 22:54:05.941406 14927 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhWOCLk/build/release/bin/kudu with pid 32010
I20260929 22:54:05.942296 32448 tablet_replica.cc:330] T be1a55688c10463dac65ea4d0f054e4b P aba31f38ad0146b8a0239028b93cb1a4: stopping tablet replica
I20260929 22:54:05.942385 32448 raft_consensus.cc:2229] T be1a55688c10463dac65ea4d0f054e4b P aba31f38ad0146b8a0239028b93cb1a4 [term 1 LEADER]: Raft consensus shutting down.
I20260929 22:54:05.942483 32448 raft_consensus.cc:2258] T be1a55688c10463dac65ea4d0f054e4b P aba31f38ad0146b8a0239028b93cb1a4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 22:54:05.942555 32449 ts_tablet_manager.cc:1905] T be1a55688c10463dac65ea4d0f054e4b P 526cb183e1eb40a8b5b5075a2621b2d9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 22:54:05.942878 32448 ts_tablet_manager.cc:1905] T be1a55688c10463dac65ea4d0f054e4b P aba31f38ad0146b8a0239028b93cb1a4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 22:54:05.944535 32449 ts_tablet_manager.cc:1918] T be1a55688c10463dac65ea4d0f054e4b P 526cb183e1eb40a8b5b5075a2621b2d9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 22:54:05.944555 32448 ts_tablet_manager.cc:1918] T be1a55688c10463dac65ea4d0f054e4b P aba31f38ad0146b8a0239028b93cb1a4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 22:54:05.944607 32449 log.cc:1196] T be1a55688c10463dac65ea4d0f054e4b P 526cb183e1eb40a8b5b5075a2621b2d9: Deleting WAL directory at /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790722213368738-14927-0/minicluster-data/ts-2/wal/wals/be1a55688c10463dac65ea4d0f054e4b
I20260929 22:54:05.944607 32448 log.cc:1196] T be1a55688c10463dac65ea4d0f054e4b P aba31f38ad0146b8a0239028b93cb1a4: Deleting WAL directory at /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790722213368738-14927-0/minicluster-data/ts-1/wal/wals/be1a55688c10463dac65ea4d0f054e4b
I20260929 22:54:05.944868 32448 ts_tablet_manager.cc:1939] T be1a55688c10463dac65ea4d0f054e4b P aba31f38ad0146b8a0239028b93cb1a4: Deleting consensus metadata
I20260929 22:54:05.944873 32449 ts_tablet_manager.cc:1939] T be1a55688c10463dac65ea4d0f054e4b P 526cb183e1eb40a8b5b5075a2621b2d9: Deleting consensus metadata
I20260929 22:54:05.945268 31935 catalog_manager.cc:4841] TS 526cb183e1eb40a8b5b5075a2621b2d9 (127.14.147.195:46053): tablet be1a55688c10463dac65ea4d0f054e4b (table test-table [id=fdb51a5dd1dc4a059788fc38db013ee3]) successfully deleted
I20260929 22:54:05.947839 31934 catalog_manager.cc:4841] TS aba31f38ad0146b8a0239028b93cb1a4 (127.14.147.194:45767): tablet be1a55688c10463dac65ea4d0f054e4b (table test-table [id=fdb51a5dd1dc4a059788fc38db013ee3]) successfully deleted
W20260929 22:54:05.948982 31935 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.14.147.193:34797 (error 108)
I20260929 22:54:05.949244 14927 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhWOCLk/build/release/bin/kudu with pid 32145
I20260929 22:54:05.949968 32422 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:54:05.949339 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.147.193:34797 (local address 127.0.0.1:35006)
0929 22:54:05.949904 (+ 565us) negotiation.cc:107] Waiting for socket to connect
0929 22:54:05.949939 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.147.193:34797: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":546}
W20260929 22:54:05.950176 31935 catalog_manager.cc:4584] TS 8614c2a12a604df584db06eca9c6d3b5 (127.14.147.193:34797): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet be1a55688c10463dac65ea4d0f054e4b: Network error: Client connection negotiation failed: client connection to 127.14.147.193:34797: connect: Connection refused (error 111)
I20260929 22:54:05.954090 14927 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhWOCLk/build/release/bin/kudu with pid 32279
I20260929 22:54:05.958140 14927 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhWOCLk/build/release/bin/kudu with pid 31916
2026-09-29T22:54:05Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/0 (1797 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/1
2026-09-29T22:54:05Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T22:54:05Z Disabled control of system clock
I20260929 22:54:05.983091 14927 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhWOCLk/build/release/bin/kudu
/tmp/dist-test-taskhWOCLk/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790722213368738-14927-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790722213368738-14927-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790722213368738-14927-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790722213368738-14927-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.147.254:40595
--webserver_interface=127.14.147.254
--webserver_port=0
--builtin_ntp_servers=127.14.147.212:40649
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.147.254:40595
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260929 22:54:06.063403 32455 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 22:54:06.063706 32455 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 22:54:06.063730 32455 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 22:54:06.064932 32455 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 22:54:06.064975 32455 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 22:54:06.064986 32455 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 22:54:06.064998 32455 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 22:54:06.065009 32455 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 22:54:06.066398 32455 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.147.212:40649
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790722213368738-14927-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790722213368738-14927-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.14.147.254:40595
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.147.254:40595
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790722213368738-14927-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.14.147.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32455
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790722213368738-14927-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 21:58:54 UTC on ccycloud.quasar-xguoti.root.comops.site
build id 4681390
I20260929 22:54:06.066648 32455 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 22:54:06.066957 32455 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 22:54:06.071820 32460 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 22:54:06.071895 32461 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 22:54:06.072052 32455 server_base.cc:1034] running on GCE node
W20260929 22:54:06.071895 32463 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 22:54:06.072378 32455 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 22:54:06.072669 32455 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 22:54:06.073844 32455 hybrid_clock.cc:648] HybridClock initialized: now 1790722446073836 us; error 37 us; skew 500 ppm
I20260929 22:54:06.075115 32455 webserver.cc:460] Webserver started at http://127.14.147.254:41925/ using document root <none> and password file <none>
I20260929 22:54:06.075354 32455 fs_manager.cc:359] Metadata directory not provided
I20260929 22:54:06.075448 32455 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 22:54:06.075577 32455 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 22:54:06.076637 32455 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790722213368738-14927-0/minicluster-data/master-0/data/instance:
uuid: "b8b18288f6c64b39a18b36faa37a49ff"
format_stamp: "Formatted at 2026-09-29 22:54:06 on dist-test-slave-98mk"
server_key: "3b15432ed26d40bdb3063ad389ac5d9d"
server_key_iv: "3a2b8749a8dc4fe77315b24702cf374b"
server_key_version: "encryptionkey@0"
I20260929 22:54:06.076984 32455 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790722213368738-14927-0/minicluster-data/master-0/wal/instance:
uuid: "b8b18288f6c64b39a18b36faa37a49ff"
format_stamp: "Formatted at 2026-09-29 22:54:06 on dist-test-slave-98mk"
server_key: "3b15432ed26d40bdb3063ad389ac5d9d"
server_key_iv: "3a2b8749a8dc4fe77315b24702cf374b"
server_key_version: "encryptionkey@0"
I20260929 22:54:06.078341 32455 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260929 22:54:06.079198 32469 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 22:54:06.079358 32455 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260929 22:54:06.079468 32455 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790722213368738-14927-0/minicluster-data/master-0/data,/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790722213368738-14927-0/minicluster-data/master-0/wal
uuid: "b8b18288f6c64b39a18b36faa37a49ff"
format_stamp: "Formatted at 2026-09-29 22:54:06 on dist-test-slave-98mk"
server_key: "3b15432ed26d40bdb3063ad389ac5d9d"
server_key_iv: "3a2b8749a8dc4fe77315b24702cf374b"
server_key_version: "encryptionkey@0"
I20260929 22:54:06.079535 32455 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790722213368738-14927-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790722213368738-14927-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790722213368738-14927-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 22:54:06.097852 32455 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 22:54:06.098173 32455 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 22:54:06.098305 32455 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 22:54:06.102061 32455 rpc_server.cc:307] RPC server started. Bound to: 127.14.147.254:40595
I20260929 22:54:06.102101 32521 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.147.254:40595 every 8 connection(s)
I20260929 22:54:06.102412 32455 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790722213368738-14927-0/minicluster-data/master-0/data/info.pb
I20260929 22:54:06.103370 32522 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 22:54:06.105580 32522 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b8b18288f6c64b39a18b36faa37a49ff: Bootstrap starting.
I20260929 22:54:06.106087 32522 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b8b18288f6c64b39a18b36faa37a49ff: Neither blocks nor log segments found. Creating new log.
I20260929 22:54:06.106320 32522 log.cc:824] T 00000000000000000000000000000000 P b8b18288f6c64b39a18b36faa37a49ff: Log is configured to *not* fsync() on all Append() calls
I20260929 22:54:06.106902 32522 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b8b18288f6c64b39a18b36faa37a49ff: No bootstrap required, opened a new log
I20260929 22:54:06.107503 14927 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhWOCLk/build/release/bin/kudu as pid 32455
I20260929 22:54:06.107595 14927 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790722213368738-14927-0/minicluster-data/master-0/wal/instance
I20260929 22:54:06.107824 14927 external_mini_cluster.cc:1428] Setting key 113f6904f8476a97992c10f9a38677b7
I20260929 22:54:06.108877 32522 raft_consensus.cc:348] T 00000000000000000000000000000000 P b8b18288f6c64b39a18b36faa37a49ff [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b8b18288f6c64b39a18b36faa37a49ff" member_type: VOTER last_known_addr { host: "127.14.147.254" port: 40595 } }
I20260929 22:54:06.109022 32522 raft_consensus.cc:374] T 00000000000000000000000000000000 P b8b18288f6c64b39a18b36faa37a49ff [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 22:54:06.109057 32522 raft_consensus.cc:729] T 00000000000000000000000000000000 P b8b18288f6c64b39a18b36faa37a49ff [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b8b18288f6c64b39a18b36faa37a49ff, State: Initialized, Role: FOLLOWER
I20260929 22:54:06.109182 32522 consensus_queue.cc:260] T 00000000000000000000000000000000 P b8b18288f6c64b39a18b36faa37a49ff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b8b18288f6c64b39a18b36faa37a49ff" member_type: VOTER last_known_addr { host: "127.14.147.254" port: 40595 } }
I20260929 22:54:06.109273 32522 raft_consensus.cc:388] T 00000000000000000000000000000000 P b8b18288f6c64b39a18b36faa37a49ff [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 22:54:06.109301 32522 raft_consensus.cc:482] T 00000000000000000000000000000000 P b8b18288f6c64b39a18b36faa37a49ff [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 22:54:06.109344 32522 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b8b18288f6c64b39a18b36faa37a49ff [term 0 FOLLOWER]: Advancing to term 1
I20260929 22:54:06.109869 32522 raft_consensus.cc:504] T 00000000000000000000000000000000 P b8b18288f6c64b39a18b36faa37a49ff [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b8b18288f6c64b39a18b36faa37a49ff" member_type: VOTER last_known_addr { host: "127.14.147.254" port: 40595 } }
I20260929 22:54:06.109954 32522 leader_election.cc:302] T 00000000000000000000000000000000 P b8b18288f6c64b39a18b36faa37a49ff [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b8b18288f6c64b39a18b36faa37a49ff; no voters:
I20260929 22:54:06.110098 32522 leader_election.cc:288] T 00000000000000000000000000000000 P b8b18288f6c64b39a18b36faa37a49ff [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 22:54:06.110155 32527 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b8b18288f6c64b39a18b36faa37a49ff [term 1 FOLLOWER]: Leader election won for term 1
I20260929 22:54:06.110292 32527 raft_consensus.cc:686] T 00000000000000000000000000000000 P b8b18288f6c64b39a18b36faa37a49ff [term 1 LEADER]: Becoming Leader. State: Replica: b8b18288f6c64b39a18b36faa37a49ff, State: Running, Role: LEADER
I20260929 22:54:06.110348 32522 sys_catalog.cc:564] T 00000000000000000000000000000000 P b8b18288f6c64b39a18b36faa37a49ff [sys.catalog]: configured and running, proceeding with master startup.
I20260929 22:54:06.110380 32527 consensus_queue.cc:237] T 00000000000000000000000000000000 P b8b18288f6c64b39a18b36faa37a49ff [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b8b18288f6c64b39a18b36faa37a49ff" member_type: VOTER last_known_addr { host: "127.14.147.254" port: 40595 } }
I20260929 22:54:06.110728 32529 sys_catalog.cc:455] T 00000000000000000000000000000000 P b8b18288f6c64b39a18b36faa37a49ff [sys.catalog]: SysCatalogTable state changed. Reason: New leader b8b18288f6c64b39a18b36faa37a49ff. Latest consensus state: current_term: 1 leader_uuid: "b8b18288f6c64b39a18b36faa37a49ff" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b8b18288f6c64b39a18b36faa37a49ff" member_type: VOTER last_known_addr { host: "127.14.147.254" port: 40595 } } }
I20260929 22:54:06.110806 32529 sys_catalog.cc:458] T 00000000000000000000000000000000 P b8b18288f6c64b39a18b36faa37a49ff [sys.catalog]: This master's current role is: LEADER
I20260929 22:54:06.110706 32528 sys_catalog.cc:455] T 00000000000000000000000000000000 P b8b18288f6c64b39a18b36faa37a49ff [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b8b18288f6c64b39a18b36faa37a49ff" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b8b18288f6c64b39a18b36faa37a49ff" member_type: VOTER last_known_addr { host: "127.14.147.254" port: 40595 } } }
I20260929 22:54:06.110971 32528 sys_catalog.cc:458] T 00000000000000000000000000000000 P b8b18288f6c64b39a18b36faa37a49ff [sys.catalog]: This master's current role is: LEADER
I20260929 22:54:06.111076 32534 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 22:54:06.111757 32534 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 22:54:06.113430 32534 catalog_manager.cc:1347] Generated new cluster ID: afc8135b0c7d4e2c8870d27740499a63
I20260929 22:54:06.113482 32534 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 22:54:06.113869 32526 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:54:06.108341 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43056 (local address 127.14.147.254:40595)
0929 22:54:06.108535 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0929 22:54:06.108543 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 22:54:06.108568 (+ 25us) server_negotiation.cc:407] Connection header received
0929 22:54:06.108847 (+ 279us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 22:54:06.108853 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 22:54:06.108907 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 22:54:06.109012 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0929 22:54:06.109268 (+ 256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:54:06.110771 (+ 1503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:54:06.111876 (+ 1105us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:54:06.112395 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:54:06.112481 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 22:54:06.113148 (+ 667us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 22:54:06.113152 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 22:54:06.113152 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 22:54:06.113354 (+ 202us) server_negotiation.cc:922] calling sasl_server_start()
0929 22:54:06.113393 (+ 39us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 22:54:06.113407 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 22:54:06.113540 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0929 22:54:06.113577 (+ 37us) server_negotiation.cc:299] Negotiation successful
0929 22:54:06.113658 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":129,"thread_start_us":80,"threads_started":1}
I20260929 22:54:06.121233 32534 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 22:54:06.121948 32534 catalog_manager.cc:1504] Loading token signing keys...
I20260929 22:54:06.136796 32534 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b8b18288f6c64b39a18b36faa37a49ff: Generated new TSK 0
I20260929 22:54:06.137004 32534 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 22:54:06.142715 14927 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhWOCLk/build/release/bin/kudu
/tmp/dist-test-taskhWOCLk/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790722213368738-14927-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790722213368738-14927-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790722213368738-14927-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790722213368738-14927-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.147.193:0
--local_ip_for_outbound_sockets=127.14.147.193
--webserver_interface=127.14.147.193
--webserver_port=0
--tserver_master_addrs=127.14.147.254:40595
--builtin_ntp_servers=127.14.147.212:40649
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260929 22:54:06.225425 32546 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 22:54:06.225661 32546 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 22:54:06.225685 32546 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 22:54:06.227162 32546 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 22:54:06.227298 32546 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 22:54:06.227350 32546 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.147.193
I20260929 22:54:06.229756 32546 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.147.212:40649
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790722213368738-14927-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790722213368738-14927-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.147.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790722213368738-14927-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.14.147.193
--webserver_port=0
--tserver_master_addrs=127.14.147.254:40595
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32546
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.147.193
--log_dir=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790722213368738-14927-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 21:58:54 UTC on ccycloud.quasar-xguoti.root.comops.site
build id 4681390
I20260929 22:54:06.230080 32546 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 22:54:06.230526 32546 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 22:54:06.236035 32552 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 22:54:06.236048 32551 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 22:54:06.236265 32554 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 22:54:06.236296 32546 server_base.cc:1034] running on GCE node
I20260929 22:54:06.236795 32546 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 22:54:06.237109 32546 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 22:54:06.238301 32546 hybrid_clock.cc:648] HybridClock initialized: now 1790722446238283 us; error 34 us; skew 500 ppm
I20260929 22:54:06.239724 32546 webserver.cc:460] Webserver started at http://127.14.147.193:42129/ using document root <none> and password file <none>
I20260929 22:54:06.239984 32546 fs_manager.cc:359] Metadata directory not provided
I20260929 22:54:06.240033 32546 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 22:54:06.240319 32546 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 22:54:06.241670 32546 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790722213368738-14927-0/minicluster-data/ts-0/data/instance:
uuid: "36e62831be924704915b4761c2a29d96"
format_stamp: "Formatted at 2026-09-29 22:54:06 on dist-test-slave-98mk"
server_key: "f6129ab75c7997baaf310299e8a407a7"
server_key_iv: "1f0c43740f72ff64cb77a6088f819573"
server_key_version: "encryptionkey@0"
I20260929 22:54:06.242014 32546 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790722213368738-14927-0/minicluster-data/ts-0/wal/instance:
uuid: "36e62831be924704915b4761c2a29d96"
format_stamp: "Formatted at 2026-09-29 22:54:06 on dist-test-slave-98mk"
server_key: "f6129ab75c7997baaf310299e8a407a7"
server_key_iv: "1f0c43740f72ff64cb77a6088f819573"
server_key_version: "encryptionkey@0"
I20260929 22:54:06.243466 32546 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260929 22:54:06.244257 32560 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 22:54:06.244405 32546 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260929 22:54:06.244489 32546 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790722213368738-14927-0/minicluster-data/ts-0/data,/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790722213368738-14927-0/minicluster-data/ts-0/wal
uuid: "36e62831be924704915b4761c2a29d96"
format_stamp: "Formatted at 2026-09-29 22:54:06 on dist-test-slave-98mk"
server_key: "f6129ab75c7997baaf310299e8a407a7"
server_key_iv: "1f0c43740f72ff64cb77a6088f819573"
server_key_version: "encryptionkey@0"
I20260929 22:54:06.244555 32546 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790722213368738-14927-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790722213368738-14927-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790722213368738-14927-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 22:54:06.256872 32546 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 22:54:06.257200 32546 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 22:54:06.257323 32546 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 22:54:06.257555 32546 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 22:54:06.257881 32546 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 22:54:06.257920 32546 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 22:54:06.257949 32546 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 22:54:06.257966 32546 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 22:54:06.264120 32546 rpc_server.cc:307] RPC server started. Bound to: 127.14.147.193:44041
I20260929 22:54:06.264171 32673 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.147.193:44041 every 8 connection(s)
I20260929 22:54:06.264518 32546 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790722213368738-14927-0/minicluster-data/ts-0/data/info.pb
I20260929 22:54:06.267966 14927 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhWOCLk/build/release/bin/kudu as pid 32546
I20260929 22:54:06.268103 14927 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790722213368738-14927-0/minicluster-data/ts-0/wal/instance
I20260929 22:54:06.268297 14927 external_mini_cluster.cc:1428] Setting key dc38b09d7653bd90851b28b3c28e2d8d
I20260929 22:54:06.269251 14927 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhWOCLk/build/release/bin/kudu
/tmp/dist-test-taskhWOCLk/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790722213368738-14927-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790722213368738-14927-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790722213368738-14927-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790722213368738-14927-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.147.194:0
--local_ip_for_outbound_sockets=127.14.147.194
--webserver_interface=127.14.147.194
--webserver_port=0
--tserver_master_addrs=127.14.147.254:40595
--builtin_ntp_servers=127.14.147.212:40649
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260929 22:54:06.270503 32526 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:54:06.265253 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.147.193:57499 (local address 127.14.147.254:40595)
0929 22:54:06.265309 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0929 22:54:06.265311 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 22:54:06.265513 (+ 202us) server_negotiation.cc:407] Connection header received
0929 22:54:06.266112 (+ 599us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 22:54:06.266114 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 22:54:06.266127 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 22:54:06.266163 (+ 36us) server_negotiation.cc:226] Negotiated authn=SASL
0929 22:54:06.266848 (+ 685us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:54:06.267374 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:54:06.268742 (+ 1368us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:54:06.269041 (+ 299us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:54:06.269097 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 22:54:06.270081 (+ 984us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 22:54:06.270084 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 22:54:06.270085 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 22:54:06.270092 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0929 22:54:06.270121 (+ 29us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 22:54:06.270127 (+ 6us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 22:54:06.270215 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0929 22:54:06.270365 (+ 150us) server_negotiation.cc:299] Negotiation successful
0929 22:54:06.270412 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260929 22:54:06.270695 32676 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:54:06.265238 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.147.254:40595 (local address 127.14.147.193:57499)
0929 22:54:06.265406 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0929 22:54:06.265440 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0929 22:54:06.265939 (+ 499us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 22:54:06.266231 (+ 292us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 22:54:06.266234 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 22:54:06.266244 (+ 10us) client_negotiation.cc:189] Negotiated authn=SASL
0929 22:54:06.266758 (+ 514us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 22:54:06.266762 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 22:54:06.267507 (+ 745us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 22:54:06.267508 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 22:54:06.268659 (+ 1151us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 22:54:06.268663 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 22:54:06.269133 (+ 470us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 22:54:06.269134 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 22:54:06.269299 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 22:54:06.269839 (+ 540us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 22:54:06.269840 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0929 22:54:06.269881 (+ 41us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 22:54:06.269914 (+ 33us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 22:54:06.270233 (+ 319us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 22:54:06.270237 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 22:54:06.270241 (+ 4us) client_negotiation.cc:769] Sending connection context
0929 22:54:06.270318 (+ 77us) client_negotiation.cc:240] Negotiation successful
0929 22:54:06.270455 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":80,"threads_started":1}
I20260929 22:54:06.271179 32674 heartbeater.cc:347] Connected to a master server at 127.14.147.254:40595
I20260929 22:54:06.271320 32674 heartbeater.cc:464] Registering TS with master...
I20260929 22:54:06.271559 32674 heartbeater.cc:511] Master 127.14.147.254:40595 requested a full tablet report, sending...
I20260929 22:54:06.272089 32486 ts_manager.cc:194] Registered new tserver with Master: 36e62831be924704915b4761c2a29d96 (127.14.147.193:44041)
I20260929 22:54:06.272876 32486 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.14.147.193:57499
W20260929 22:54:06.350037 32677 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 22:54:06.350260 32677 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 22:54:06.350282 32677 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 22:54:06.351516 32677 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 22:54:06.351562 32677 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 22:54:06.351590 32677 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.147.194
I20260929 22:54:06.352967 32677 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.147.212:40649
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790722213368738-14927-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790722213368738-14927-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.147.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790722213368738-14927-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.14.147.194
--webserver_port=0
--tserver_master_addrs=127.14.147.254:40595
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32677
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.147.194
--log_dir=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790722213368738-14927-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 21:58:54 UTC on ccycloud.quasar-xguoti.root.comops.site
build id 4681390
I20260929 22:54:06.353180 32677 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 22:54:06.353457 32677 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 22:54:06.358467 32683 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 22:54:06.358536 32677 server_base.cc:1034] running on GCE node
W20260929 22:54:06.358467 32685 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 22:54:06.358726 32682 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 22:54:06.358952 32677 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 22:54:06.359201 32677 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 22:54:06.360353 32677 hybrid_clock.cc:648] HybridClock initialized: now 1790722446360322 us; error 47 us; skew 500 ppm
I20260929 22:54:06.361824 32677 webserver.cc:460] Webserver started at http://127.14.147.194:43503/ using document root <none> and password file <none>
I20260929 22:54:06.362061 32677 fs_manager.cc:359] Metadata directory not provided
I20260929 22:54:06.362110 32677 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 22:54:06.362257 32677 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 22:54:06.363384 32677 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790722213368738-14927-0/minicluster-data/ts-1/data/instance:
uuid: "207fbcc661994a5188df5b9932b4e9fd"
format_stamp: "Formatted at 2026-09-29 22:54:06 on dist-test-slave-98mk"
server_key: "2b1a1e22c5e3b663df501242e2a7979b"
server_key_iv: "2626f22b41184c8b034567f727a11fd8"
server_key_version: "encryptionkey@0"
I20260929 22:54:06.363775 32677 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790722213368738-14927-0/minicluster-data/ts-1/wal/instance:
uuid: "207fbcc661994a5188df5b9932b4e9fd"
format_stamp: "Formatted at 2026-09-29 22:54:06 on dist-test-slave-98mk"
server_key: "2b1a1e22c5e3b663df501242e2a7979b"
server_key_iv: "2626f22b41184c8b034567f727a11fd8"
server_key_version: "encryptionkey@0"
I20260929 22:54:06.365276 32677 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260929 22:54:06.366072 32691 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 22:54:06.366366 32677 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260929 22:54:06.366516 32677 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790722213368738-14927-0/minicluster-data/ts-1/data,/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790722213368738-14927-0/minicluster-data/ts-1/wal
uuid: "207fbcc661994a5188df5b9932b4e9fd"
format_stamp: "Formatted at 2026-09-29 22:54:06 on dist-test-slave-98mk"
server_key: "2b1a1e22c5e3b663df501242e2a7979b"
server_key_iv: "2626f22b41184c8b034567f727a11fd8"
server_key_version: "encryptionkey@0"
I20260929 22:54:06.366632 32677 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790722213368738-14927-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790722213368738-14927-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790722213368738-14927-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 22:54:06.381271 32677 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 22:54:06.381610 32677 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 22:54:06.381748 32677 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 22:54:06.381974 32677 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 22:54:06.382331 32677 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 22:54:06.382364 32677 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 22:54:06.382393 32677 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 22:54:06.382411 32677 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 22:54:06.387966 32677 rpc_server.cc:307] RPC server started. Bound to: 127.14.147.194:42307
I20260929 22:54:06.388020 336 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.147.194:42307 every 8 connection(s)
I20260929 22:54:06.388348 32677 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790722213368738-14927-0/minicluster-data/ts-1/data/info.pb
I20260929 22:54:06.393589 32526 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:54:06.389052 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.147.194:33707 (local address 127.14.147.254:40595)
0929 22:54:06.389112 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0929 22:54:06.389114 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 22:54:06.389320 (+ 206us) server_negotiation.cc:407] Connection header received
0929 22:54:06.389804 (+ 484us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 22:54:06.389805 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 22:54:06.389814 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 22:54:06.389886 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0929 22:54:06.390572 (+ 686us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:54:06.391092 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:54:06.392135 (+ 1043us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:54:06.392321 (+ 186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:54:06.392378 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 22:54:06.393335 (+ 957us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 22:54:06.393336 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 22:54:06.393337 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 22:54:06.393342 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0929 22:54:06.393359 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 22:54:06.393364 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 22:54:06.393408 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0929 22:54:06.393486 (+ 78us) server_negotiation.cc:299] Negotiation successful
0929 22:54:06.393519 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260929 22:54:06.393784 339 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:54:06.389035 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.147.254:40595 (local address 127.14.147.194:33707)
0929 22:54:06.389211 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0929 22:54:06.389254 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0929 22:54:06.389726 (+ 472us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 22:54:06.389942 (+ 216us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 22:54:06.389945 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 22:54:06.389954 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
0929 22:54:06.390464 (+ 510us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 22:54:06.390468 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 22:54:06.391210 (+ 742us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 22:54:06.391211 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 22:54:06.392065 (+ 854us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 22:54:06.392068 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 22:54:06.392556 (+ 488us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 22:54:06.392561 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 22:54:06.392693 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 22:54:06.393188 (+ 495us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 22:54:06.393189 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0929 22:54:06.393218 (+ 29us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 22:54:06.393245 (+ 27us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 22:54:06.393426 (+ 181us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 22:54:06.393429 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 22:54:06.393432 (+ 3us) client_negotiation.cc:769] Sending connection context
0929 22:54:06.393481 (+ 49us) client_negotiation.cc:240] Negotiation successful
0929 22:54:06.393565 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":68,"threads_started":1}
I20260929 22:54:06.394264 337 heartbeater.cc:347] Connected to a master server at 127.14.147.254:40595
I20260929 22:54:06.394364 337 heartbeater.cc:464] Registering TS with master...
I20260929 22:54:06.394523 337 heartbeater.cc:511] Master 127.14.147.254:40595 requested a full tablet report, sending...
I20260929 22:54:06.394914 32486 ts_manager.cc:194] Registered new tserver with Master: 207fbcc661994a5188df5b9932b4e9fd (127.14.147.194:42307)
I20260929 22:54:06.395036 14927 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhWOCLk/build/release/bin/kudu as pid 32677
I20260929 22:54:06.395123 14927 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790722213368738-14927-0/minicluster-data/ts-1/wal/instance
I20260929 22:54:06.395323 14927 external_mini_cluster.cc:1428] Setting key 01303408efc99c49f57a3868c88dbdb1
I20260929 22:54:06.395447 32486 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.14.147.194:33707
I20260929 22:54:06.396184 14927 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhWOCLk/build/release/bin/kudu
/tmp/dist-test-taskhWOCLk/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790722213368738-14927-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790722213368738-14927-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790722213368738-14927-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790722213368738-14927-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.147.195:0
--local_ip_for_outbound_sockets=127.14.147.195
--webserver_interface=127.14.147.195
--webserver_port=0
--tserver_master_addrs=127.14.147.254:40595
--builtin_ntp_servers=127.14.147.212:40649
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260929 22:54:06.475934 340 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 22:54:06.476174 340 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 22:54:06.476207 340 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 22:54:06.477447 340 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 22:54:06.477495 340 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 22:54:06.477532 340 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.147.195
I20260929 22:54:06.478945 340 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.147.212:40649
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790722213368738-14927-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790722213368738-14927-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.147.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790722213368738-14927-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.14.147.195
--webserver_port=0
--tserver_master_addrs=127.14.147.254:40595
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.340
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.147.195
--log_dir=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790722213368738-14927-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 21:58:54 UTC on ccycloud.quasar-xguoti.root.comops.site
build id 4681390
I20260929 22:54:06.479207 340 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 22:54:06.479545 340 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260929 22:54:06.484627 340 server_base.cc:1034] running on GCE node
W20260929 22:54:06.484731 348 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 22:54:06.484781 345 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 22:54:06.484702 346 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 22:54:06.485015 340 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 22:54:06.485271 340 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 22:54:06.486438 340 hybrid_clock.cc:648] HybridClock initialized: now 1790722446486437 us; error 44 us; skew 500 ppm
I20260929 22:54:06.487807 340 webserver.cc:460] Webserver started at http://127.14.147.195:45861/ using document root <none> and password file <none>
I20260929 22:54:06.488041 340 fs_manager.cc:359] Metadata directory not provided
I20260929 22:54:06.488091 340 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 22:54:06.488219 340 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 22:54:06.489293 340 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790722213368738-14927-0/minicluster-data/ts-2/data/instance:
uuid: "e9f3633cfcaf4c71a683d997fb330f30"
format_stamp: "Formatted at 2026-09-29 22:54:06 on dist-test-slave-98mk"
server_key: "c1d8220ec8112e05cacf5f3670a2782a"
server_key_iv: "f94dbfb9195318307b9e76d402e6303a"
server_key_version: "encryptionkey@0"
I20260929 22:54:06.489631 340 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790722213368738-14927-0/minicluster-data/ts-2/wal/instance:
uuid: "e9f3633cfcaf4c71a683d997fb330f30"
format_stamp: "Formatted at 2026-09-29 22:54:06 on dist-test-slave-98mk"
server_key: "c1d8220ec8112e05cacf5f3670a2782a"
server_key_iv: "f94dbfb9195318307b9e76d402e6303a"
server_key_version: "encryptionkey@0"
I20260929 22:54:06.490882 340 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260929 22:54:06.491631 354 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 22:54:06.491775 340 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260929 22:54:06.491863 340 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790722213368738-14927-0/minicluster-data/ts-2/data,/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790722213368738-14927-0/minicluster-data/ts-2/wal
uuid: "e9f3633cfcaf4c71a683d997fb330f30"
format_stamp: "Formatted at 2026-09-29 22:54:06 on dist-test-slave-98mk"
server_key: "c1d8220ec8112e05cacf5f3670a2782a"
server_key_iv: "f94dbfb9195318307b9e76d402e6303a"
server_key_version: "encryptionkey@0"
I20260929 22:54:06.491928 340 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790722213368738-14927-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790722213368738-14927-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790722213368738-14927-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 22:54:06.518719 340 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 22:54:06.519034 340 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 22:54:06.519161 340 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 22:54:06.519384 340 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 22:54:06.519734 340 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 22:54:06.519767 340 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 22:54:06.519796 340 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 22:54:06.519819 340 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 22:54:06.525113 340 rpc_server.cc:307] RPC server started. Bound to: 127.14.147.195:33957
I20260929 22:54:06.525166 467 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.147.195:33957 every 8 connection(s)
I20260929 22:54:06.525486 340 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790722213368738-14927-0/minicluster-data/ts-2/data/info.pb
I20260929 22:54:06.530520 32526 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:54:06.526201 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.147.195:50757 (local address 127.14.147.254:40595)
0929 22:54:06.526262 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0929 22:54:06.526264 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 22:54:06.526494 (+ 230us) server_negotiation.cc:407] Connection header received
0929 22:54:06.527015 (+ 521us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 22:54:06.527016 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 22:54:06.527029 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 22:54:06.527077 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0929 22:54:06.527792 (+ 715us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:54:06.528289 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:54:06.529232 (+ 943us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:54:06.529425 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:54:06.529479 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 22:54:06.530257 (+ 778us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 22:54:06.530258 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 22:54:06.530259 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 22:54:06.530263 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0929 22:54:06.530279 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 22:54:06.530282 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 22:54:06.530323 (+ 41us) server_negotiation.cc:1035] Waiting for connection context
0929 22:54:06.530404 (+ 81us) server_negotiation.cc:299] Negotiation successful
0929 22:54:06.530436 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260929 22:54:06.530709 470 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:54:06.526199 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.147.254:40595 (local address 127.14.147.195:50757)
0929 22:54:06.526375 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0929 22:54:06.526412 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0929 22:54:06.526920 (+ 508us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 22:54:06.527156 (+ 236us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 22:54:06.527159 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 22:54:06.527167 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0929 22:54:06.527708 (+ 541us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 22:54:06.527714 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 22:54:06.528375 (+ 661us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 22:54:06.528375 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 22:54:06.529153 (+ 778us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 22:54:06.529155 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 22:54:06.529499 (+ 344us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 22:54:06.529500 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 22:54:06.529624 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 22:54:06.530125 (+ 501us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 22:54:06.530125 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0929 22:54:06.530154 (+ 29us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 22:54:06.530178 (+ 24us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 22:54:06.530329 (+ 151us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 22:54:06.530335 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 22:54:06.530338 (+ 3us) client_negotiation.cc:769] Sending connection context
0929 22:54:06.530396 (+ 58us) client_negotiation.cc:240] Negotiation successful
0929 22:54:06.530497 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":72,"threads_started":1}
I20260929 22:54:06.530982 14927 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhWOCLk/build/release/bin/kudu as pid 340
I20260929 22:54:06.531071 14927 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790722213368738-14927-0/minicluster-data/ts-2/wal/instance
I20260929 22:54:06.531216 468 heartbeater.cc:347] Connected to a master server at 127.14.147.254:40595
I20260929 22:54:06.531287 14927 external_mini_cluster.cc:1428] Setting key ebf20824e23b042fe0e5751c5a885200
I20260929 22:54:06.531311 468 heartbeater.cc:464] Registering TS with master...
I20260929 22:54:06.531526 468 heartbeater.cc:511] Master 127.14.147.254:40595 requested a full tablet report, sending...
I20260929 22:54:06.531900 32486 ts_manager.cc:194] Registered new tserver with Master: e9f3633cfcaf4c71a683d997fb330f30 (127.14.147.195:33957)
I20260929 22:54:06.532265 14927 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 22:54:06.532316 32486 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.14.147.195:50757
I20260929 22:54:06.536154 32526 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:54:06.533666 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43072 (local address 127.14.147.254:40595)
0929 22:54:06.533728 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0929 22:54:06.533730 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 22:54:06.533841 (+ 111us) server_negotiation.cc:407] Connection header received
0929 22:54:06.533866 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 22:54:06.533866 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 22:54:06.533876 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 22:54:06.533920 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0929 22:54:06.534024 (+ 104us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:54:06.534475 (+ 451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:54:06.535194 (+ 719us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:54:06.535365 (+ 171us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:54:06.535405 (+ 40us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 22:54:06.535997 (+ 592us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 22:54:06.535997 (+ 0us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 22:54:06.535998 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 22:54:06.536001 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0929 22:54:06.536011 (+ 10us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 22:54:06.536014 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 22:54:06.536048 (+ 34us) server_negotiation.cc:1035] Waiting for connection context
0929 22:54:06.536079 (+ 31us) server_negotiation.cc:299] Negotiation successful
0929 22:54:06.536097 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260929 22:54:06.537592 32486 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:43072:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 22:54:06.539126 32486 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 22:54:06.545905 477 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:54:06.541832 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.147.193:44041 (local address 127.0.0.1:43442)
0929 22:54:06.542241 (+ 409us) negotiation.cc:107] Waiting for socket to connect
0929 22:54:06.542249 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0929 22:54:06.542281 (+ 32us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 22:54:06.542513 (+ 232us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 22:54:06.542514 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 22:54:06.542516 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 22:54:06.542566 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 22:54:06.542568 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 22:54:06.543938 (+ 1370us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 22:54:06.543939 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 22:54:06.544900 (+ 961us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 22:54:06.544904 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 22:54:06.545587 (+ 683us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 22:54:06.545588 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 22:54:06.545669 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 22:54:06.545683 (+ 14us) client_negotiation.cc:769] Sending connection context
0929 22:54:06.545746 (+ 63us) client_negotiation.cc:240] Negotiation successful
0929 22:54:06.545800 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":394,"mutex_wait_us":55,"thread_start_us":125,"threads_started":1}
I20260929 22:54:06.545912 478 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:54:06.541830 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.147.195:33957 (local address 127.0.0.1:42118)
0929 22:54:06.542031 (+ 201us) negotiation.cc:107] Waiting for socket to connect
0929 22:54:06.542071 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0929 22:54:06.542138 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 22:54:06.542444 (+ 306us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 22:54:06.542446 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 22:54:06.542449 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 22:54:06.542522 (+ 73us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 22:54:06.542524 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 22:54:06.543515 (+ 991us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 22:54:06.543516 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 22:54:06.544851 (+ 1335us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 22:54:06.544854 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 22:54:06.545732 (+ 878us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 22:54:06.545732 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 22:54:06.545787 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 22:54:06.545790 (+ 3us) client_negotiation.cc:769] Sending connection context
0929 22:54:06.545826 (+ 36us) client_negotiation.cc:240] Negotiation successful
0929 22:54:06.545858 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":170,"thread_start_us":64,"threads_started":1}
I20260929 22:54:06.545996 480 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:54:06.542074 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42118 (local address 127.14.147.195:33957)
0929 22:54:06.542247 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0929 22:54:06.542252 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 22:54:06.542263 (+ 11us) server_negotiation.cc:407] Connection header received
0929 22:54:06.542310 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 22:54:06.542312 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 22:54:06.542328 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 22:54:06.542418 (+ 90us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 22:54:06.542587 (+ 169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:54:06.543452 (+ 865us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:54:06.544931 (+ 1479us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:54:06.545663 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:54:06.545748 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 22:54:06.545838 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0929 22:54:06.545864 (+ 26us) server_negotiation.cc:299] Negotiation successful
0929 22:54:06.545906 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":134,"thread_start_us":77,"threads_started":1}
I20260929 22:54:06.546263 479 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:54:06.541933 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43442 (local address 127.14.147.193:44041)
0929 22:54:06.542369 (+ 436us) server_negotiation.cc:206] Beginning negotiation
0929 22:54:06.542372 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 22:54:06.542380 (+ 8us) server_negotiation.cc:407] Connection header received
0929 22:54:06.542416 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 22:54:06.542418 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 22:54:06.542433 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 22:54:06.542496 (+ 63us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 22:54:06.542646 (+ 150us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:54:06.543834 (+ 1188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:54:06.544971 (+ 1137us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:54:06.545510 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:54:06.545576 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 22:54:06.545671 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0929 22:54:06.546139 (+ 468us) server_negotiation.cc:299] Negotiation successful
0929 22:54:06.546186 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":403,"thread_start_us":71,"threads_started":1}
I20260929 22:54:06.546516 481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:54:06.542098 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49202 (local address 127.14.147.194:42307)
0929 22:54:06.542302 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0929 22:54:06.542305 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 22:54:06.542475 (+ 170us) server_negotiation.cc:407] Connection header received
0929 22:54:06.542516 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 22:54:06.542518 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 22:54:06.542532 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 22:54:06.542595 (+ 63us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 22:54:06.542751 (+ 156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:54:06.543859 (+ 1108us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:54:06.545061 (+ 1202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:54:06.545741 (+ 680us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:54:06.545800 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 22:54:06.545870 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0929 22:54:06.546392 (+ 522us) server_negotiation.cc:299] Negotiation successful
0929 22:54:06.546439 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":169,"thread_start_us":83,"threads_started":1}
I20260929 22:54:06.546648 476 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:54:06.541830 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.147.194:42307 (local address 127.0.0.1:49202)
0929 22:54:06.542363 (+ 533us) negotiation.cc:107] Waiting for socket to connect
0929 22:54:06.542373 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0929 22:54:06.542426 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 22:54:06.542653 (+ 227us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 22:54:06.542655 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 22:54:06.542657 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 22:54:06.542686 (+ 29us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 22:54:06.542687 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 22:54:06.543935 (+ 1248us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 22:54:06.543936 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 22:54:06.544990 (+ 1054us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 22:54:06.544992 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 22:54:06.545904 (+ 912us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 22:54:06.545905 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 22:54:06.545972 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 22:54:06.546164 (+ 192us) client_negotiation.cc:769] Sending connection context
0929 22:54:06.546214 (+ 50us) client_negotiation.cc:240] Negotiation successful
0929 22:54:06.546271 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":510,"mutex_wait_us":12,"thread_start_us":88,"threads_started":1}
I20260929 22:54:06.546689 402 tablet_service.cc:1467] Processing CreateTablet for tablet 073e330cc2b0460b9dbb92a718903e72 (DEFAULT_TABLE table=test-table [id=29e54f75ba0446f8b8193af3b99cbc86]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 22:54:06.547089 402 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 073e330cc2b0460b9dbb92a718903e72. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 22:54:06.547004 32608 tablet_service.cc:1467] Processing CreateTablet for tablet 073e330cc2b0460b9dbb92a718903e72 (DEFAULT_TABLE table=test-table [id=29e54f75ba0446f8b8193af3b99cbc86]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 22:54:06.547278 32608 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 073e330cc2b0460b9dbb92a718903e72. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 22:54:06.549840 482 tablet_bootstrap.cc:492] T 073e330cc2b0460b9dbb92a718903e72 P 36e62831be924704915b4761c2a29d96: Bootstrap starting.
I20260929 22:54:06.550359 32739 tablet_service.cc:1467] Processing CreateTablet for tablet 073e330cc2b0460b9dbb92a718903e72 (DEFAULT_TABLE table=test-table [id=29e54f75ba0446f8b8193af3b99cbc86]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 22:54:06.550532 482 tablet_bootstrap.cc:654] T 073e330cc2b0460b9dbb92a718903e72 P 36e62831be924704915b4761c2a29d96: Neither blocks nor log segments found. Creating new log.
I20260929 22:54:06.550696 32739 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 073e330cc2b0460b9dbb92a718903e72. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 22:54:06.550832 482 log.cc:824] T 073e330cc2b0460b9dbb92a718903e72 P 36e62831be924704915b4761c2a29d96: Log is configured to *not* fsync() on all Append() calls
I20260929 22:54:06.551158 483 tablet_bootstrap.cc:492] T 073e330cc2b0460b9dbb92a718903e72 P e9f3633cfcaf4c71a683d997fb330f30: Bootstrap starting.
I20260929 22:54:06.551602 482 tablet_bootstrap.cc:492] T 073e330cc2b0460b9dbb92a718903e72 P 36e62831be924704915b4761c2a29d96: No bootstrap required, opened a new log
I20260929 22:54:06.551662 482 ts_tablet_manager.cc:1397] T 073e330cc2b0460b9dbb92a718903e72 P 36e62831be924704915b4761c2a29d96: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260929 22:54:06.553345 485 tablet_bootstrap.cc:492] T 073e330cc2b0460b9dbb92a718903e72 P 207fbcc661994a5188df5b9932b4e9fd: Bootstrap starting.
I20260929 22:54:06.553448 482 raft_consensus.cc:348] T 073e330cc2b0460b9dbb92a718903e72 P 36e62831be924704915b4761c2a29d96 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9f3633cfcaf4c71a683d997fb330f30" member_type: VOTER last_known_addr { host: "127.14.147.195" port: 33957 } } peers { permanent_uuid: "36e62831be924704915b4761c2a29d96" member_type: VOTER last_known_addr { host: "127.14.147.193" port: 44041 } } peers { permanent_uuid: "207fbcc661994a5188df5b9932b4e9fd" member_type: VOTER last_known_addr { host: "127.14.147.194" port: 42307 } }
I20260929 22:54:06.553619 482 raft_consensus.cc:374] T 073e330cc2b0460b9dbb92a718903e72 P 36e62831be924704915b4761c2a29d96 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 22:54:06.553644 482 raft_consensus.cc:729] T 073e330cc2b0460b9dbb92a718903e72 P 36e62831be924704915b4761c2a29d96 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 36e62831be924704915b4761c2a29d96, State: Initialized, Role: FOLLOWER
I20260929 22:54:06.553757 482 consensus_queue.cc:260] T 073e330cc2b0460b9dbb92a718903e72 P 36e62831be924704915b4761c2a29d96 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9f3633cfcaf4c71a683d997fb330f30" member_type: VOTER last_known_addr { host: "127.14.147.195" port: 33957 } } peers { permanent_uuid: "36e62831be924704915b4761c2a29d96" member_type: VOTER last_known_addr { host: "127.14.147.193" port: 44041 } } peers { permanent_uuid: "207fbcc661994a5188df5b9932b4e9fd" member_type: VOTER last_known_addr { host: "127.14.147.194" port: 42307 } }
I20260929 22:54:06.553989 485 tablet_bootstrap.cc:654] T 073e330cc2b0460b9dbb92a718903e72 P 207fbcc661994a5188df5b9932b4e9fd: Neither blocks nor log segments found. Creating new log.
I20260929 22:54:06.554011 482 ts_tablet_manager.cc:1428] T 073e330cc2b0460b9dbb92a718903e72 P 36e62831be924704915b4761c2a29d96: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260929 22:54:06.554282 485 log.cc:824] T 073e330cc2b0460b9dbb92a718903e72 P 207fbcc661994a5188df5b9932b4e9fd: Log is configured to *not* fsync() on all Append() calls
I20260929 22:54:06.554574 483 tablet_bootstrap.cc:654] T 073e330cc2b0460b9dbb92a718903e72 P e9f3633cfcaf4c71a683d997fb330f30: Neither blocks nor log segments found. Creating new log.
I20260929 22:54:06.554859 483 log.cc:824] T 073e330cc2b0460b9dbb92a718903e72 P e9f3633cfcaf4c71a683d997fb330f30: Log is configured to *not* fsync() on all Append() calls
I20260929 22:54:06.554944 485 tablet_bootstrap.cc:492] T 073e330cc2b0460b9dbb92a718903e72 P 207fbcc661994a5188df5b9932b4e9fd: No bootstrap required, opened a new log
I20260929 22:54:06.555008 485 ts_tablet_manager.cc:1397] T 073e330cc2b0460b9dbb92a718903e72 P 207fbcc661994a5188df5b9932b4e9fd: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260929 22:54:06.555100 32674 heartbeater.cc:503] Master 127.14.147.254:40595 was elected leader, sending a full tablet report...
I20260929 22:54:06.555719 483 tablet_bootstrap.cc:492] T 073e330cc2b0460b9dbb92a718903e72 P e9f3633cfcaf4c71a683d997fb330f30: No bootstrap required, opened a new log
I20260929 22:54:06.555788 483 ts_tablet_manager.cc:1397] T 073e330cc2b0460b9dbb92a718903e72 P e9f3633cfcaf4c71a683d997fb330f30: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.000s
I20260929 22:54:06.556912 485 raft_consensus.cc:348] T 073e330cc2b0460b9dbb92a718903e72 P 207fbcc661994a5188df5b9932b4e9fd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9f3633cfcaf4c71a683d997fb330f30" member_type: VOTER last_known_addr { host: "127.14.147.195" port: 33957 } } peers { permanent_uuid: "36e62831be924704915b4761c2a29d96" member_type: VOTER last_known_addr { host: "127.14.147.193" port: 44041 } } peers { permanent_uuid: "207fbcc661994a5188df5b9932b4e9fd" member_type: VOTER last_known_addr { host: "127.14.147.194" port: 42307 } }
I20260929 22:54:06.557042 485 raft_consensus.cc:374] T 073e330cc2b0460b9dbb92a718903e72 P 207fbcc661994a5188df5b9932b4e9fd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 22:54:06.557072 485 raft_consensus.cc:729] T 073e330cc2b0460b9dbb92a718903e72 P 207fbcc661994a5188df5b9932b4e9fd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 207fbcc661994a5188df5b9932b4e9fd, State: Initialized, Role: FOLLOWER
I20260929 22:54:06.557185 485 consensus_queue.cc:260] T 073e330cc2b0460b9dbb92a718903e72 P 207fbcc661994a5188df5b9932b4e9fd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9f3633cfcaf4c71a683d997fb330f30" member_type: VOTER last_known_addr { host: "127.14.147.195" port: 33957 } } peers { permanent_uuid: "36e62831be924704915b4761c2a29d96" member_type: VOTER last_known_addr { host: "127.14.147.193" port: 44041 } } peers { permanent_uuid: "207fbcc661994a5188df5b9932b4e9fd" member_type: VOTER last_known_addr { host: "127.14.147.194" port: 42307 } }
I20260929 22:54:06.557432 485 ts_tablet_manager.cc:1428] T 073e330cc2b0460b9dbb92a718903e72 P 207fbcc661994a5188df5b9932b4e9fd: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260929 22:54:06.557479 337 heartbeater.cc:503] Master 127.14.147.254:40595 was elected leader, sending a full tablet report...
I20260929 22:54:06.557833 483 raft_consensus.cc:348] T 073e330cc2b0460b9dbb92a718903e72 P e9f3633cfcaf4c71a683d997fb330f30 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9f3633cfcaf4c71a683d997fb330f30" member_type: VOTER last_known_addr { host: "127.14.147.195" port: 33957 } } peers { permanent_uuid: "36e62831be924704915b4761c2a29d96" member_type: VOTER last_known_addr { host: "127.14.147.193" port: 44041 } } peers { permanent_uuid: "207fbcc661994a5188df5b9932b4e9fd" member_type: VOTER last_known_addr { host: "127.14.147.194" port: 42307 } }
I20260929 22:54:06.557986 483 raft_consensus.cc:374] T 073e330cc2b0460b9dbb92a718903e72 P e9f3633cfcaf4c71a683d997fb330f30 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 22:54:06.558022 483 raft_consensus.cc:729] T 073e330cc2b0460b9dbb92a718903e72 P e9f3633cfcaf4c71a683d997fb330f30 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e9f3633cfcaf4c71a683d997fb330f30, State: Initialized, Role: FOLLOWER
I20260929 22:54:06.558130 483 consensus_queue.cc:260] T 073e330cc2b0460b9dbb92a718903e72 P e9f3633cfcaf4c71a683d997fb330f30 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9f3633cfcaf4c71a683d997fb330f30" member_type: VOTER last_known_addr { host: "127.14.147.195" port: 33957 } } peers { permanent_uuid: "36e62831be924704915b4761c2a29d96" member_type: VOTER last_known_addr { host: "127.14.147.193" port: 44041 } } peers { permanent_uuid: "207fbcc661994a5188df5b9932b4e9fd" member_type: VOTER last_known_addr { host: "127.14.147.194" port: 42307 } }
I20260929 22:54:06.558368 483 ts_tablet_manager.cc:1428] T 073e330cc2b0460b9dbb92a718903e72 P e9f3633cfcaf4c71a683d997fb330f30: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260929 22:54:06.558481 468 heartbeater.cc:503] Master 127.14.147.254:40595 was elected leader, sending a full tablet report...
I20260929 22:54:06.627193 486 raft_consensus.cc:482] T 073e330cc2b0460b9dbb92a718903e72 P 36e62831be924704915b4761c2a29d96 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 22:54:06.627346 486 raft_consensus.cc:504] T 073e330cc2b0460b9dbb92a718903e72 P 36e62831be924704915b4761c2a29d96 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9f3633cfcaf4c71a683d997fb330f30" member_type: VOTER last_known_addr { host: "127.14.147.195" port: 33957 } } peers { permanent_uuid: "36e62831be924704915b4761c2a29d96" member_type: VOTER last_known_addr { host: "127.14.147.193" port: 44041 } } peers { permanent_uuid: "207fbcc661994a5188df5b9932b4e9fd" member_type: VOTER last_known_addr { host: "127.14.147.194" port: 42307 } }
I20260929 22:54:06.627753 486 leader_election.cc:288] T 073e330cc2b0460b9dbb92a718903e72 P 36e62831be924704915b4761c2a29d96 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e9f3633cfcaf4c71a683d997fb330f30 (127.14.147.195:33957), 207fbcc661994a5188df5b9932b4e9fd (127.14.147.194:42307)
I20260929 22:54:06.630509 480 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:54:06.628007 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.147.193:38801 (local address 127.14.147.195:33957)
0929 22:54:06.628057 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0929 22:54:06.628059 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 22:54:06.628207 (+ 148us) server_negotiation.cc:407] Connection header received
0929 22:54:06.628258 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 22:54:06.628259 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 22:54:06.628271 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 22:54:06.628313 (+ 42us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 22:54:06.628446 (+ 133us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:54:06.628921 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:54:06.629833 (+ 912us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:54:06.630194 (+ 361us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:54:06.630258 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 22:54:06.630266 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0929 22:54:06.630395 (+ 129us) server_negotiation.cc:299] Negotiation successful
0929 22:54:06.630432 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260929 22:54:06.630510 481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:54:06.628046 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.147.193:52623 (local address 127.14.147.194:42307)
0929 22:54:06.628087 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0929 22:54:06.628089 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 22:54:06.628097 (+ 8us) server_negotiation.cc:407] Connection header received
0929 22:54:06.628126 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 22:54:06.628127 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 22:54:06.628134 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 22:54:06.628200 (+ 66us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 22:54:06.628353 (+ 153us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:54:06.628830 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:54:06.629804 (+ 974us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:54:06.630168 (+ 364us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:54:06.630249 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 22:54:06.630259 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0929 22:54:06.630392 (+ 133us) server_negotiation.cc:299] Negotiation successful
0929 22:54:06.630436 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260929 22:54:06.630530 491 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:54:06.627936 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.147.195:33957 (local address 127.14.147.193:38801)
0929 22:54:06.628145 (+ 209us) negotiation.cc:107] Waiting for socket to connect
0929 22:54:06.628152 (+ 7us) client_negotiation.cc:174] Beginning negotiation
0929 22:54:06.628194 (+ 42us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 22:54:06.628320 (+ 126us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 22:54:06.628321 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 22:54:06.628323 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 22:54:06.628388 (+ 65us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 22:54:06.628389 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 22:54:06.628993 (+ 604us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 22:54:06.628994 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 22:54:06.629748 (+ 754us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 22:54:06.629751 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 22:54:06.630265 (+ 514us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 22:54:06.630266 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 22:54:06.630336 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 22:54:06.630339 (+ 3us) client_negotiation.cc:769] Sending connection context
0929 22:54:06.630381 (+ 42us) client_negotiation.cc:240] Negotiation successful
0929 22:54:06.630422 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":195,"mutex_wait_us":59,"thread_start_us":104,"threads_started":1}
I20260929 22:54:06.630532 32676 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:54:06.627931 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.147.194:42307 (local address 127.14.147.193:52623)
0929 22:54:06.627978 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0929 22:54:06.627991 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 22:54:06.628043 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 22:54:06.628207 (+ 164us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 22:54:06.628209 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 22:54:06.628211 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 22:54:06.628284 (+ 73us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 22:54:06.628286 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 22:54:06.628909 (+ 623us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 22:54:06.628911 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 22:54:06.629703 (+ 792us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 22:54:06.629707 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 22:54:06.630258 (+ 551us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 22:54:06.630259 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 22:54:06.630325 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 22:54:06.630329 (+ 4us) client_negotiation.cc:769] Sending connection context
0929 22:54:06.630374 (+ 45us) client_negotiation.cc:240] Negotiation successful
0929 22:54:06.630422 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":27}
I20260929 22:54:06.630893 422 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "073e330cc2b0460b9dbb92a718903e72" candidate_uuid: "36e62831be924704915b4761c2a29d96" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e9f3633cfcaf4c71a683d997fb330f30" is_pre_election: true
I20260929 22:54:06.630914 32759 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "073e330cc2b0460b9dbb92a718903e72" candidate_uuid: "36e62831be924704915b4761c2a29d96" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "207fbcc661994a5188df5b9932b4e9fd" is_pre_election: true
I20260929 22:54:06.631062 32759 raft_consensus.cc:2454] T 073e330cc2b0460b9dbb92a718903e72 P 207fbcc661994a5188df5b9932b4e9fd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 36e62831be924704915b4761c2a29d96 in term 0.
I20260929 22:54:06.631062 422 raft_consensus.cc:2454] T 073e330cc2b0460b9dbb92a718903e72 P e9f3633cfcaf4c71a683d997fb330f30 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 36e62831be924704915b4761c2a29d96 in term 0.
I20260929 22:54:06.631316 32562 leader_election.cc:302] T 073e330cc2b0460b9dbb92a718903e72 P 36e62831be924704915b4761c2a29d96 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 36e62831be924704915b4761c2a29d96, e9f3633cfcaf4c71a683d997fb330f30; no voters:
I20260929 22:54:06.631511 486 raft_consensus.cc:2781] T 073e330cc2b0460b9dbb92a718903e72 P 36e62831be924704915b4761c2a29d96 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 22:54:06.631553 486 raft_consensus.cc:482] T 073e330cc2b0460b9dbb92a718903e72 P 36e62831be924704915b4761c2a29d96 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 22:54:06.631574 486 raft_consensus.cc:3037] T 073e330cc2b0460b9dbb92a718903e72 P 36e62831be924704915b4761c2a29d96 [term 0 FOLLOWER]: Advancing to term 1
I20260929 22:54:06.632378 486 raft_consensus.cc:504] T 073e330cc2b0460b9dbb92a718903e72 P 36e62831be924704915b4761c2a29d96 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9f3633cfcaf4c71a683d997fb330f30" member_type: VOTER last_known_addr { host: "127.14.147.195" port: 33957 } } peers { permanent_uuid: "36e62831be924704915b4761c2a29d96" member_type: VOTER last_known_addr { host: "127.14.147.193" port: 44041 } } peers { permanent_uuid: "207fbcc661994a5188df5b9932b4e9fd" member_type: VOTER last_known_addr { host: "127.14.147.194" port: 42307 } }
I20260929 22:54:06.632530 486 leader_election.cc:288] T 073e330cc2b0460b9dbb92a718903e72 P 36e62831be924704915b4761c2a29d96 [CANDIDATE]: Term 1 election: Requested vote from peers e9f3633cfcaf4c71a683d997fb330f30 (127.14.147.195:33957), 207fbcc661994a5188df5b9932b4e9fd (127.14.147.194:42307)
I20260929 22:54:06.632710 422 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "073e330cc2b0460b9dbb92a718903e72" candidate_uuid: "36e62831be924704915b4761c2a29d96" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e9f3633cfcaf4c71a683d997fb330f30"
I20260929 22:54:06.632791 32759 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "073e330cc2b0460b9dbb92a718903e72" candidate_uuid: "36e62831be924704915b4761c2a29d96" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "207fbcc661994a5188df5b9932b4e9fd"
I20260929 22:54:06.632800 422 raft_consensus.cc:3037] T 073e330cc2b0460b9dbb92a718903e72 P e9f3633cfcaf4c71a683d997fb330f30 [term 0 FOLLOWER]: Advancing to term 1
I20260929 22:54:06.632871 32759 raft_consensus.cc:3037] T 073e330cc2b0460b9dbb92a718903e72 P 207fbcc661994a5188df5b9932b4e9fd [term 0 FOLLOWER]: Advancing to term 1
I20260929 22:54:06.633775 32759 raft_consensus.cc:2454] T 073e330cc2b0460b9dbb92a718903e72 P 207fbcc661994a5188df5b9932b4e9fd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 36e62831be924704915b4761c2a29d96 in term 1.
I20260929 22:54:06.633776 422 raft_consensus.cc:2454] T 073e330cc2b0460b9dbb92a718903e72 P e9f3633cfcaf4c71a683d997fb330f30 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 36e62831be924704915b4761c2a29d96 in term 1.
I20260929 22:54:06.633960 32561 leader_election.cc:302] T 073e330cc2b0460b9dbb92a718903e72 P 36e62831be924704915b4761c2a29d96 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 207fbcc661994a5188df5b9932b4e9fd, 36e62831be924704915b4761c2a29d96; no voters:
I20260929 22:54:06.634050 486 raft_consensus.cc:2781] T 073e330cc2b0460b9dbb92a718903e72 P 36e62831be924704915b4761c2a29d96 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 22:54:06.634188 486 raft_consensus.cc:686] T 073e330cc2b0460b9dbb92a718903e72 P 36e62831be924704915b4761c2a29d96 [term 1 LEADER]: Becoming Leader. State: Replica: 36e62831be924704915b4761c2a29d96, State: Running, Role: LEADER
I20260929 22:54:06.634274 486 consensus_queue.cc:237] T 073e330cc2b0460b9dbb92a718903e72 P 36e62831be924704915b4761c2a29d96 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9f3633cfcaf4c71a683d997fb330f30" member_type: VOTER last_known_addr { host: "127.14.147.195" port: 33957 } } peers { permanent_uuid: "36e62831be924704915b4761c2a29d96" member_type: VOTER last_known_addr { host: "127.14.147.193" port: 44041 } } peers { permanent_uuid: "207fbcc661994a5188df5b9932b4e9fd" member_type: VOTER last_known_addr { host: "127.14.147.194" port: 42307 } }
I20260929 22:54:06.635051 32486 catalog_manager.cc:5495] T 073e330cc2b0460b9dbb92a718903e72 P 36e62831be924704915b4761c2a29d96 reported cstate change: term changed from 0 to 1, leader changed from <none> to 36e62831be924704915b4761c2a29d96 (127.14.147.193). New cstate: current_term: 1 leader_uuid: "36e62831be924704915b4761c2a29d96" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9f3633cfcaf4c71a683d997fb330f30" member_type: VOTER last_known_addr { host: "127.14.147.195" port: 33957 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "36e62831be924704915b4761c2a29d96" member_type: VOTER last_known_addr { host: "127.14.147.193" port: 44041 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "207fbcc661994a5188df5b9932b4e9fd" member_type: VOTER last_known_addr { host: "127.14.147.194" port: 42307 } health_report { overall_health: UNKNOWN } } }
W20260929 22:54:06.638882 338 tablet.cc:2369] T 073e330cc2b0460b9dbb92a718903e72 P 207fbcc661994a5188df5b9932b4e9fd: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 22:54:06.662537 479 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:54:06.660267 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43448 (local address 127.14.147.193:44041)
0929 22:54:06.660331 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0929 22:54:06.660333 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 22:54:06.660340 (+ 7us) server_negotiation.cc:407] Connection header received
0929 22:54:06.660389 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 22:54:06.660390 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 22:54:06.660409 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 22:54:06.660478 (+ 69us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 22:54:06.660604 (+ 126us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:54:06.661095 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:54:06.661820 (+ 725us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:54:06.662007 (+ 187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:54:06.662062 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 22:54:06.662211 (+ 149us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 22:54:06.662319 (+ 108us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 22:54:06.662398 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0929 22:54:06.662418 (+ 20us) server_negotiation.cc:299] Negotiation successful
0929 22:54:06.662440 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260929 22:54:06.664393 32759 raft_consensus.cc:1261] T 073e330cc2b0460b9dbb92a718903e72 P 207fbcc661994a5188df5b9932b4e9fd [term 1 FOLLOWER]: Refusing update from remote peer 36e62831be924704915b4761c2a29d96: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 22:54:06.664394 422 raft_consensus.cc:1261] T 073e330cc2b0460b9dbb92a718903e72 P e9f3633cfcaf4c71a683d997fb330f30 [term 1 FOLLOWER]: Refusing update from remote peer 36e62831be924704915b4761c2a29d96: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 22:54:06.664758 492 consensus_queue.cc:1035] T 073e330cc2b0460b9dbb92a718903e72 P 36e62831be924704915b4761c2a29d96 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e9f3633cfcaf4c71a683d997fb330f30" member_type: VOTER last_known_addr { host: "127.14.147.195" port: 33957 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 22:54:06.664892 486 consensus_queue.cc:1035] T 073e330cc2b0460b9dbb92a718903e72 P 36e62831be924704915b4761c2a29d96 [LEADER]: Connected to new peer: Peer: permanent_uuid: "207fbcc661994a5188df5b9932b4e9fd" member_type: VOTER last_known_addr { host: "127.14.147.194" port: 42307 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 22:54:06.667207 495 mvcc.cc:205] Tried to move back new op lower bound from 7334799141534711808 to 7334799141414617088. Current Snapshot: MvccSnapshot[applied={T|T < 7334799141534711808}]
I20260929 22:54:06.667754 496 mvcc.cc:205] Tried to move back new op lower bound from 7334799141534711808 to 7334799141414617088. Current Snapshot: MvccSnapshot[applied={T|T < 7334799141534711808}]
I20260929 22:54:06.669665 32486 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:43072:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 22:54:06.669766 32486 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:43072:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 22:54:06.670604 32486 catalog_manager.cc:5782] T 00000000000000000000000000000000 P b8b18288f6c64b39a18b36faa37a49ff: Sending DeleteTablet for 3 replicas of tablet 073e330cc2b0460b9dbb92a718903e72
I20260929 22:54:06.670907 32608 tablet_service.cc:1514] Processing DeleteTablet for tablet 073e330cc2b0460b9dbb92a718903e72 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 22:54:06 UTC) from {username='slave'} at 127.0.0.1:43442
I20260929 22:54:06.670974 402 tablet_service.cc:1514] Processing DeleteTablet for tablet 073e330cc2b0460b9dbb92a718903e72 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 22:54:06 UTC) from {username='slave'} at 127.0.0.1:42118
I20260929 22:54:06.671170 505 tablet_replica.cc:330] T 073e330cc2b0460b9dbb92a718903e72 P 36e62831be924704915b4761c2a29d96: stopping tablet replica
I20260929 22:54:06.671168 32739 tablet_service.cc:1514] Processing DeleteTablet for tablet 073e330cc2b0460b9dbb92a718903e72 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 22:54:06 UTC) from {username='slave'} at 127.0.0.1:49202
I20260929 22:54:06.671277 505 raft_consensus.cc:2229] T 073e330cc2b0460b9dbb92a718903e72 P 36e62831be924704915b4761c2a29d96 [term 1 LEADER]: Raft consensus shutting down.
I20260929 22:54:06.671370 505 raft_consensus.cc:2258] T 073e330cc2b0460b9dbb92a718903e72 P 36e62831be924704915b4761c2a29d96 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 22:54:06.671384 507 tablet_replica.cc:330] T 073e330cc2b0460b9dbb92a718903e72 P 207fbcc661994a5188df5b9932b4e9fd: stopping tablet replica
I20260929 22:54:06.671634 507 raft_consensus.cc:2229] T 073e330cc2b0460b9dbb92a718903e72 P 207fbcc661994a5188df5b9932b4e9fd [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 22:54:06.671748 507 raft_consensus.cc:2258] T 073e330cc2b0460b9dbb92a718903e72 P 207fbcc661994a5188df5b9932b4e9fd [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 22:54:06.671754 14927 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhWOCLk/build/release/bin/kudu with pid 32546
I20260929 22:54:06.671772 506 tablet_replica.cc:330] T 073e330cc2b0460b9dbb92a718903e72 P e9f3633cfcaf4c71a683d997fb330f30: stopping tablet replica
I20260929 22:54:06.671944 506 raft_consensus.cc:2229] T 073e330cc2b0460b9dbb92a718903e72 P e9f3633cfcaf4c71a683d997fb330f30 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 22:54:06.672078 506 raft_consensus.cc:2258] T 073e330cc2b0460b9dbb92a718903e72 P e9f3633cfcaf4c71a683d997fb330f30 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 22:54:06.672603 507 ts_tablet_manager.cc:1905] T 073e330cc2b0460b9dbb92a718903e72 P 207fbcc661994a5188df5b9932b4e9fd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 22:54:06.672603 506 ts_tablet_manager.cc:1905] T 073e330cc2b0460b9dbb92a718903e72 P e9f3633cfcaf4c71a683d997fb330f30: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 22:54:06.674365 506 ts_tablet_manager.cc:1918] T 073e330cc2b0460b9dbb92a718903e72 P e9f3633cfcaf4c71a683d997fb330f30: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 22:54:06.674367 507 ts_tablet_manager.cc:1918] T 073e330cc2b0460b9dbb92a718903e72 P 207fbcc661994a5188df5b9932b4e9fd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 22:54:06.674444 507 log.cc:1196] T 073e330cc2b0460b9dbb92a718903e72 P 207fbcc661994a5188df5b9932b4e9fd: Deleting WAL directory at /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790722213368738-14927-0/minicluster-data/ts-1/wal/wals/073e330cc2b0460b9dbb92a718903e72
I20260929 22:54:06.674444 506 log.cc:1196] T 073e330cc2b0460b9dbb92a718903e72 P e9f3633cfcaf4c71a683d997fb330f30: Deleting WAL directory at /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790722213368738-14927-0/minicluster-data/ts-2/wal/wals/073e330cc2b0460b9dbb92a718903e72
I20260929 22:54:06.674687 507 ts_tablet_manager.cc:1939] T 073e330cc2b0460b9dbb92a718903e72 P 207fbcc661994a5188df5b9932b4e9fd: Deleting consensus metadata
I20260929 22:54:06.674687 506 ts_tablet_manager.cc:1939] T 073e330cc2b0460b9dbb92a718903e72 P e9f3633cfcaf4c71a683d997fb330f30: Deleting consensus metadata
I20260929 22:54:06.675100 32471 catalog_manager.cc:4841] TS e9f3633cfcaf4c71a683d997fb330f30 (127.14.147.195:33957): tablet 073e330cc2b0460b9dbb92a718903e72 (table test-table [id=29e54f75ba0446f8b8193af3b99cbc86]) successfully deleted
I20260929 22:54:06.675160 32470 catalog_manager.cc:4841] TS 207fbcc661994a5188df5b9932b4e9fd (127.14.147.194:42307): tablet 073e330cc2b0460b9dbb92a718903e72 (table test-table [id=29e54f75ba0446f8b8193af3b99cbc86]) successfully deleted
W20260929 22:54:06.678946 32473 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.14.147.193:44041 (error 108)
I20260929 22:54:06.679042 14927 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhWOCLk/build/release/bin/kudu with pid 32677
I20260929 22:54:06.680354 478 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:54:06.680217 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.147.193:44041 (local address 127.0.0.1:43462)
0929 22:54:06.680274 (+ 57us) negotiation.cc:107] Waiting for socket to connect
0929 22:54:06.680317 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.147.193:44041: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
W20260929 22:54:06.680502 32473 catalog_manager.cc:4584] TS 36e62831be924704915b4761c2a29d96 (127.14.147.193:44041): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 073e330cc2b0460b9dbb92a718903e72: Network error: Client connection negotiation failed: client connection to 127.14.147.193:44041: connect: Connection refused (error 111)
I20260929 22:54:06.683933 14927 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhWOCLk/build/release/bin/kudu with pid 340
I20260929 22:54:06.687736 14927 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhWOCLk/build/release/bin/kudu with pid 32455
2026-09-29T22:54:06Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/1 (729 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/2
Loading random data
Initializing database '/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790722213368738-14927-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 22:54:06 dist-test-slave-98mk krb5kdc[511](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 22:54:06 dist-test-slave-98mk krb5kdc[511](info): set up 2 sockets
Sep 29 22:54:06 dist-test-slave-98mk krb5kdc[511](info): commencing operation
krb5kdc: starting...
W20260929 22:54:07.639767 14927 mini_kdc.cc:121] Time spent starting KDC: real 0.934s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790722213368738-14927-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790722213368738-14927-0/krb5kdc/test-user.keytab.
Sep 29 22:54:07 dist-test-slave-98mk krb5kdc[511](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790722447, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T22:54:07Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T22:54:07Z Disabled control of system clock
WARNING: no policy specified for kudu/127.14.147.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.147.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.147.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790722213368738-14927-0/krb5kdc/kudu_127.14.147.254.keytab.
Entry for principal kudu/127.14.147.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790722213368738-14927-0/krb5kdc/kudu_127.14.147.254.keytab.
I20260929 22:54:07.753739 14927 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhWOCLk/build/release/bin/kudu
/tmp/dist-test-taskhWOCLk/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790722213368738-14927-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790722213368738-14927-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790722213368738-14927-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790722213368738-14927-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.147.254:34313
--webserver_interface=127.14.147.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790722213368738-14927-0/krb5kdc/kudu_127.14.147.254.keytab
--principal=kudu/127.14.147.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.147.212:44843
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.147.254:34313
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790722213368738-14927-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790722213368738-14927-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790722213368738-14927-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 22:54:07.834590 524 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 22:54:07.834811 524 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 22:54:07.834842 524 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 22:54:07.836088 524 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 22:54:07.836130 524 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 22:54:07.836143 524 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 22:54:07.836154 524 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 22:54:07.836165 524 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 22:54:07.837663 524 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.147.212:44843
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790722213368738-14927-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790722213368738-14927-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.14.147.254:34313
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790722213368738-14927-0/krb5kdc/kudu_127.14.147.254.keytab
--principal=kudu/127.14.147.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.147.254:34313
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790722213368738-14927-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.14.147.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.524
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790722213368738-14927-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 21:58:54 UTC on ccycloud.quasar-xguoti.root.comops.site
build id 4681390
I20260929 22:54:07.837891 524 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 22:54:07.838076 524 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 22:54:07.842721 532 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 22:54:07.842710 529 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 22:54:07.842823 524 server_base.cc:1034] running on GCE node
W20260929 22:54:07.842715 530 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 22:54:07.843237 524 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 22:54:07.843554 524 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 22:54:07.844702 524 hybrid_clock.cc:648] HybridClock initialized: now 1790722447844681 us; error 38 us; skew 500 ppm
Sep 29 22:54:07 dist-test-slave-98mk krb5kdc[511](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790722447, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.147.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 22:54:07.847187 524 init.cc:377] Logged in from keytab as kudu/127.14.147.254@KRBTEST.COM (short username kudu)
I20260929 22:54:07.847648 524 webserver.cc:460] Webserver started at http://127.14.147.254:43019/ using document root <none> and password file <none>
I20260929 22:54:07.847851 524 fs_manager.cc:359] Metadata directory not provided
I20260929 22:54:07.847893 524 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 22:54:07.847977 524 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 22:54:07.848999 524 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790722213368738-14927-0/minicluster-data/master-0/data/instance:
uuid: "750ef0065148452b93d625c887ae243a"
format_stamp: "Formatted at 2026-09-29 22:54:07 on dist-test-slave-98mk"
server_key: "e7cb0688f049338a2ceec4be25173908"
server_key_iv: "37657c9e539f928ec2c7b658b0191d39"
server_key_version: "encryptionkey@0"
I20260929 22:54:07.849283 524 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790722213368738-14927-0/minicluster-data/master-0/wal/instance:
uuid: "750ef0065148452b93d625c887ae243a"
format_stamp: "Formatted at 2026-09-29 22:54:07 on dist-test-slave-98mk"
server_key: "e7cb0688f049338a2ceec4be25173908"
server_key_iv: "37657c9e539f928ec2c7b658b0191d39"
server_key_version: "encryptionkey@0"
I20260929 22:54:07.850453 524 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260929 22:54:07.851085 539 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 22:54:07.851248 524 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260929 22:54:07.851332 524 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790722213368738-14927-0/minicluster-data/master-0/data,/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790722213368738-14927-0/minicluster-data/master-0/wal
uuid: "750ef0065148452b93d625c887ae243a"
format_stamp: "Formatted at 2026-09-29 22:54:07 on dist-test-slave-98mk"
server_key: "e7cb0688f049338a2ceec4be25173908"
server_key_iv: "37657c9e539f928ec2c7b658b0191d39"
server_key_version: "encryptionkey@0"
I20260929 22:54:07.851392 524 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790722213368738-14927-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790722213368738-14927-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790722213368738-14927-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 22:54:07.866809 524 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 22:54:07.869313 524 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 22:54:07.869444 524 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 22:54:07.873740 524 rpc_server.cc:307] RPC server started. Bound to: 127.14.147.254:34313
I20260929 22:54:07.873776 591 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.147.254:34313 every 8 connection(s)
I20260929 22:54:07.874141 524 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790722213368738-14927-0/minicluster-data/master-0/data/info.pb
I20260929 22:54:07.875054 593 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 22:54:07.877445 593 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 750ef0065148452b93d625c887ae243a: Bootstrap starting.
I20260929 22:54:07.878023 593 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 750ef0065148452b93d625c887ae243a: Neither blocks nor log segments found. Creating new log.
I20260929 22:54:07.878274 593 log.cc:824] T 00000000000000000000000000000000 P 750ef0065148452b93d625c887ae243a: Log is configured to *not* fsync() on all Append() calls
I20260929 22:54:07.878834 14927 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhWOCLk/build/release/bin/kudu as pid 524
I20260929 22:54:07.878954 14927 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790722213368738-14927-0/minicluster-data/master-0/wal/instance
I20260929 22:54:07.879109 593 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 750ef0065148452b93d625c887ae243a: No bootstrap required, opened a new log
I20260929 22:54:07.879161 14927 external_mini_cluster.cc:1428] Setting key cde12ca2da6319a006c4ee940f3d1322
I20260929 22:54:07.881529 593 raft_consensus.cc:348] T 00000000000000000000000000000000 P 750ef0065148452b93d625c887ae243a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "750ef0065148452b93d625c887ae243a" member_type: VOTER last_known_addr { host: "127.14.147.254" port: 34313 } }
I20260929 22:54:07.881682 593 raft_consensus.cc:374] T 00000000000000000000000000000000 P 750ef0065148452b93d625c887ae243a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 22:54:07.881712 593 raft_consensus.cc:729] T 00000000000000000000000000000000 P 750ef0065148452b93d625c887ae243a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 750ef0065148452b93d625c887ae243a, State: Initialized, Role: FOLLOWER
I20260929 22:54:07.881834 593 consensus_queue.cc:260] T 00000000000000000000000000000000 P 750ef0065148452b93d625c887ae243a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "750ef0065148452b93d625c887ae243a" member_type: VOTER last_known_addr { host: "127.14.147.254" port: 34313 } }
I20260929 22:54:07.881904 593 raft_consensus.cc:388] T 00000000000000000000000000000000 P 750ef0065148452b93d625c887ae243a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 22:54:07.881937 593 raft_consensus.cc:482] T 00000000000000000000000000000000 P 750ef0065148452b93d625c887ae243a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 22:54:07.881982 593 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 750ef0065148452b93d625c887ae243a [term 0 FOLLOWER]: Advancing to term 1
I20260929 22:54:07.882586 593 raft_consensus.cc:504] T 00000000000000000000000000000000 P 750ef0065148452b93d625c887ae243a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "750ef0065148452b93d625c887ae243a" member_type: VOTER last_known_addr { host: "127.14.147.254" port: 34313 } }
I20260929 22:54:07.882683 593 leader_election.cc:302] T 00000000000000000000000000000000 P 750ef0065148452b93d625c887ae243a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 750ef0065148452b93d625c887ae243a; no voters:
I20260929 22:54:07.882850 593 leader_election.cc:288] T 00000000000000000000000000000000 P 750ef0065148452b93d625c887ae243a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 22:54:07.882901 598 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 750ef0065148452b93d625c887ae243a [term 1 FOLLOWER]: Leader election won for term 1
I20260929 22:54:07.883081 598 raft_consensus.cc:686] T 00000000000000000000000000000000 P 750ef0065148452b93d625c887ae243a [term 1 LEADER]: Becoming Leader. State: Replica: 750ef0065148452b93d625c887ae243a, State: Running, Role: LEADER
I20260929 22:54:07.883117 593 sys_catalog.cc:564] T 00000000000000000000000000000000 P 750ef0065148452b93d625c887ae243a [sys.catalog]: configured and running, proceeding with master startup.
I20260929 22:54:07.883198 598 consensus_queue.cc:237] T 00000000000000000000000000000000 P 750ef0065148452b93d625c887ae243a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "750ef0065148452b93d625c887ae243a" member_type: VOTER last_known_addr { host: "127.14.147.254" port: 34313 } }
I20260929 22:54:07.883549 598 sys_catalog.cc:455] T 00000000000000000000000000000000 P 750ef0065148452b93d625c887ae243a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 750ef0065148452b93d625c887ae243a. Latest consensus state: current_term: 1 leader_uuid: "750ef0065148452b93d625c887ae243a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "750ef0065148452b93d625c887ae243a" member_type: VOTER last_known_addr { host: "127.14.147.254" port: 34313 } } }
I20260929 22:54:07.883670 598 sys_catalog.cc:458] T 00000000000000000000000000000000 P 750ef0065148452b93d625c887ae243a [sys.catalog]: This master's current role is: LEADER
I20260929 22:54:07.883558 600 sys_catalog.cc:455] T 00000000000000000000000000000000 P 750ef0065148452b93d625c887ae243a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "750ef0065148452b93d625c887ae243a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "750ef0065148452b93d625c887ae243a" member_type: VOTER last_known_addr { host: "127.14.147.254" port: 34313 } } }
I20260929 22:54:07.883749 600 sys_catalog.cc:458] T 00000000000000000000000000000000 P 750ef0065148452b93d625c887ae243a [sys.catalog]: This master's current role is: LEADER
W20260929 22:54:07.884963 613 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 750ef0065148452b93d625c887ae243a: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 22:54:07.885051 613 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260929 22:54:07.885133 605 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 22:54:07.885332 605 catalog_manager.cc:1484] Initializing Kudu cluster ID...
Sep 29 22:54:07 dist-test-slave-98mk krb5kdc[511](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790722447, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.147.254@KRBTEST.COM
I20260929 22:54:07.887003 605 catalog_manager.cc:1347] Generated new cluster ID: eea95fcc68da47e59ec0178415d7b547
I20260929 22:54:07.887056 605 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 22:54:07.890846 597 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:54:07.879742 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45532 (local address 127.14.147.254:34313)
0929 22:54:07.879948 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0929 22:54:07.879956 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 22:54:07.879982 (+ 26us) server_negotiation.cc:407] Connection header received
0929 22:54:07.880502 (+ 520us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 22:54:07.880510 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 22:54:07.880562 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 22:54:07.880679 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0929 22:54:07.881510 (+ 831us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:54:07.882740 (+ 1230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:54:07.883815 (+ 1075us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:54:07.884309 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:54:07.884431 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 22:54:07.887175 (+ 2744us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 22:54:07.887179 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 22:54:07.887179 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 22:54:07.887191 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0929 22:54:07.889149 (+ 1958us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 22:54:07.889561 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 22:54:07.889564 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 22:54:07.889564 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 22:54:07.889677 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 22:54:07.889853 (+ 176us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 22:54:07.889853 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 22:54:07.889854 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 22:54:07.890178 (+ 324us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 22:54:07.890337 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0929 22:54:07.890461 (+ 124us) server_negotiation.cc:299] Negotiation successful
0929 22:54:07.890612 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":154,"thread_start_us":100,"threads_started":1}
I20260929 22:54:07.891949 605 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 22:54:07.892463 605 catalog_manager.cc:1504] Loading token signing keys...
I20260929 22:54:07.895108 605 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 750ef0065148452b93d625c887ae243a: Generated new TSK 0
I20260929 22:54:07.895268 605 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.14.147.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.147.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.147.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790722213368738-14927-0/krb5kdc/kudu_127.14.147.193.keytab.
Entry for principal kudu/127.14.147.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790722213368738-14927-0/krb5kdc/kudu_127.14.147.193.keytab.
I20260929 22:54:07.925634 14927 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhWOCLk/build/release/bin/kudu
/tmp/dist-test-taskhWOCLk/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790722213368738-14927-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790722213368738-14927-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790722213368738-14927-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790722213368738-14927-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.147.193:0
--local_ip_for_outbound_sockets=127.14.147.193
--webserver_interface=127.14.147.193
--webserver_port=0
--tserver_master_addrs=127.14.147.254:34313
--keytab_file=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790722213368738-14927-0/krb5kdc/kudu_127.14.147.193.keytab
--principal=kudu/127.14.147.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.147.212:44843
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790722213368738-14927-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790722213368738-14927-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790722213368738-14927-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 22:54:08.006793 619 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 22:54:08.007001 619 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 22:54:08.007026 619 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 22:54:08.008270 619 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 22:54:08.008329 619 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 22:54:08.008358 619 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.147.193
I20260929 22:54:08.009892 619 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.147.212:44843
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790722213368738-14927-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790722213368738-14927-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790722213368738-14927-0/krb5kdc/kudu_127.14.147.193.keytab
--principal=kudu/127.14.147.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.147.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790722213368738-14927-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.14.147.193
--webserver_port=0
--tserver_master_addrs=127.14.147.254:34313
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.619
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.147.193
--log_dir=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790722213368738-14927-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 21:58:54 UTC on ccycloud.quasar-xguoti.root.comops.site
build id 4681390
I20260929 22:54:08.010146 619 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 22:54:08.010377 619 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 22:54:08.015141 625 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 22:54:08.015286 619 server_base.cc:1034] running on GCE node
W20260929 22:54:08.015147 627 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 22:54:08.015159 624 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 22:54:08.015616 619 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 22:54:08.015837 619 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 22:54:08.016975 619 hybrid_clock.cc:648] HybridClock initialized: now 1790722448016956 us; error 36 us; skew 500 ppm
Sep 29 22:54:08 dist-test-slave-98mk krb5kdc[511](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790722448, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.147.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 22:54:08.019376 619 init.cc:377] Logged in from keytab as kudu/127.14.147.193@KRBTEST.COM (short username kudu)
I20260929 22:54:08.019834 619 webserver.cc:460] Webserver started at http://127.14.147.193:35613/ using document root <none> and password file <none>
I20260929 22:54:08.020056 619 fs_manager.cc:359] Metadata directory not provided
I20260929 22:54:08.020103 619 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 22:54:08.020223 619 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 22:54:08.021245 619 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790722213368738-14927-0/minicluster-data/ts-0/data/instance:
uuid: "5ca5d8fcb5ea447ebc9554162325cc77"
format_stamp: "Formatted at 2026-09-29 22:54:08 on dist-test-slave-98mk"
server_key: "85e5c1b54f75ea2a64af5c5380bb455e"
server_key_iv: "966ae52bae4c13b29a2d09cc11053efd"
server_key_version: "encryptionkey@0"
I20260929 22:54:08.021575 619 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790722213368738-14927-0/minicluster-data/ts-0/wal/instance:
uuid: "5ca5d8fcb5ea447ebc9554162325cc77"
format_stamp: "Formatted at 2026-09-29 22:54:08 on dist-test-slave-98mk"
server_key: "85e5c1b54f75ea2a64af5c5380bb455e"
server_key_iv: "966ae52bae4c13b29a2d09cc11053efd"
server_key_version: "encryptionkey@0"
I20260929 22:54:08.022943 619 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260929 22:54:08.023762 634 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 22:54:08.023926 619 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260929 22:54:08.024010 619 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790722213368738-14927-0/minicluster-data/ts-0/data,/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790722213368738-14927-0/minicluster-data/ts-0/wal
uuid: "5ca5d8fcb5ea447ebc9554162325cc77"
format_stamp: "Formatted at 2026-09-29 22:54:08 on dist-test-slave-98mk"
server_key: "85e5c1b54f75ea2a64af5c5380bb455e"
server_key_iv: "966ae52bae4c13b29a2d09cc11053efd"
server_key_version: "encryptionkey@0"
I20260929 22:54:08.024075 619 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790722213368738-14927-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790722213368738-14927-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790722213368738-14927-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 22:54:08.039753 619 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 22:54:08.041872 619 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 22:54:08.042008 619 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 22:54:08.042222 619 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 22:54:08.042523 619 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 22:54:08.042557 619 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 22:54:08.042585 619 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 22:54:08.042608 619 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 22:54:08.048169 619 rpc_server.cc:307] RPC server started. Bound to: 127.14.147.193:42539
I20260929 22:54:08.048224 747 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.147.193:42539 every 8 connection(s)
I20260929 22:54:08.048564 619 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790722213368738-14927-0/minicluster-data/ts-0/data/info.pb
I20260929 22:54:08.050611 14927 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhWOCLk/build/release/bin/kudu as pid 619
I20260929 22:54:08.050714 14927 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790722213368738-14927-0/minicluster-data/ts-0/wal/instance
I20260929 22:54:08.050890 14927 external_mini_cluster.cc:1428] Setting key afcfeb9f655fc0004e857679aa916f74
Sep 29 22:54:08 dist-test-slave-98mk krb5kdc[511](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790722448, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.147.193@KRBTEST.COM for kudu/127.14.147.254@KRBTEST.COM
I20260929 22:54:08.058568 597 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:54:08.049259 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.147.193:52793 (local address 127.14.147.254:34313)
0929 22:54:08.049308 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0929 22:54:08.049310 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 22:54:08.049541 (+ 231us) server_negotiation.cc:407] Connection header received
0929 22:54:08.050007 (+ 466us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 22:54:08.050008 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 22:54:08.050017 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 22:54:08.050068 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0929 22:54:08.051034 (+ 966us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:54:08.051740 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:54:08.052734 (+ 994us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:54:08.052949 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:54:08.053004 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 22:54:08.055454 (+ 2450us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 22:54:08.055456 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 22:54:08.055456 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 22:54:08.055464 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0929 22:54:08.056828 (+ 1364us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 22:54:08.057337 (+ 509us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 22:54:08.057338 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 22:54:08.057338 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 22:54:08.057414 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 22:54:08.057729 (+ 315us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 22:54:08.057730 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 22:54:08.057730 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 22:54:08.057952 (+ 222us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 22:54:08.058019 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0929 22:54:08.058397 (+ 378us) server_negotiation.cc:299] Negotiation successful
0929 22:54:08.058491 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
WARNING: no policy specified for kudu/127.14.147.194@KRBTEST.COM; defaulting to no policy
I20260929 22:54:08.058784 750 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:54:08.049232 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.147.254:34313 (local address 127.14.147.193:52793)
0929 22:54:08.049428 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0929 22:54:08.049465 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0929 22:54:08.049933 (+ 468us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 22:54:08.050122 (+ 189us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 22:54:08.050125 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 22:54:08.050678 (+ 553us) client_negotiation.cc:189] Negotiated authn=SASL
0929 22:54:08.050906 (+ 228us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 22:54:08.050912 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 22:54:08.051840 (+ 928us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 22:54:08.051841 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 22:54:08.052652 (+ 811us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 22:54:08.052655 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 22:54:08.053022 (+ 367us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 22:54:08.053023 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 22:54:08.053149 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 22:54:08.053680 (+ 531us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 22:54:08.053690 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0929 22:54:08.055334 (+ 1644us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 22:54:08.056917 (+ 1583us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 22:54:08.056926 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 22:54:08.056928 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 22:54:08.057238 (+ 310us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 22:54:08.057486 (+ 248us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 22:54:08.057486 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 22:54:08.057487 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 22:54:08.057623 (+ 136us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 22:54:08.058030 (+ 407us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 22:54:08.058037 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 22:54:08.058222 (+ 185us) client_negotiation.cc:769] Sending connection context
0929 22:54:08.058360 (+ 138us) client_negotiation.cc:240] Negotiation successful
0929 22:54:08.058543 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":157,"thread_start_us":115,"threads_started":1}
I20260929 22:54:08.059376 748 heartbeater.cc:347] Connected to a master server at 127.14.147.254:34313
I20260929 22:54:08.059507 748 heartbeater.cc:464] Registering TS with master...
I20260929 22:54:08.059670 748 heartbeater.cc:511] Master 127.14.147.254:34313 requested a full tablet report, sending...
I20260929 22:54:08.060266 556 ts_manager.cc:194] Registered new tserver with Master: 5ca5d8fcb5ea447ebc9554162325cc77 (127.14.147.193:42539)
I20260929 22:54:08.060878 556 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.147.193@KRBTEST.COM'} at 127.14.147.193:52793
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.147.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.147.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790722213368738-14927-0/krb5kdc/kudu_127.14.147.194.keytab.
Entry for principal kudu/127.14.147.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790722213368738-14927-0/krb5kdc/kudu_127.14.147.194.keytab.
I20260929 22:54:08.075560 14927 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhWOCLk/build/release/bin/kudu
/tmp/dist-test-taskhWOCLk/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790722213368738-14927-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790722213368738-14927-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790722213368738-14927-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790722213368738-14927-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.147.194:0
--local_ip_for_outbound_sockets=127.14.147.194
--webserver_interface=127.14.147.194
--webserver_port=0
--tserver_master_addrs=127.14.147.254:34313
--keytab_file=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790722213368738-14927-0/krb5kdc/kudu_127.14.147.194.keytab
--principal=kudu/127.14.147.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.147.212:44843
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790722213368738-14927-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790722213368738-14927-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790722213368738-14927-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 22:54:08.157850 753 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 22:54:08.158069 753 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 22:54:08.158094 753 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 22:54:08.159258 753 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 22:54:08.159307 753 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 22:54:08.159335 753 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.147.194
I20260929 22:54:08.160914 753 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.147.212:44843
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790722213368738-14927-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790722213368738-14927-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790722213368738-14927-0/krb5kdc/kudu_127.14.147.194.keytab
--principal=kudu/127.14.147.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.147.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790722213368738-14927-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.14.147.194
--webserver_port=0
--tserver_master_addrs=127.14.147.254:34313
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.753
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.147.194
--log_dir=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790722213368738-14927-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 21:58:54 UTC on ccycloud.quasar-xguoti.root.comops.site
build id 4681390
I20260929 22:54:08.161131 753 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 22:54:08.161334 753 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 22:54:08.166070 759 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 22:54:08.166076 761 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 22:54:08.166226 758 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 22:54:08.166260 753 server_base.cc:1034] running on GCE node
I20260929 22:54:08.166550 753 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 22:54:08.166750 753 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 22:54:08.167887 753 hybrid_clock.cc:648] HybridClock initialized: now 1790722448167870 us; error 31 us; skew 500 ppm
Sep 29 22:54:08 dist-test-slave-98mk krb5kdc[511](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790722448, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.147.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 22:54:08.170691 753 init.cc:377] Logged in from keytab as kudu/127.14.147.194@KRBTEST.COM (short username kudu)
I20260929 22:54:08.171173 753 webserver.cc:460] Webserver started at http://127.14.147.194:40811/ using document root <none> and password file <none>
I20260929 22:54:08.171409 753 fs_manager.cc:359] Metadata directory not provided
I20260929 22:54:08.171496 753 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 22:54:08.171629 753 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 22:54:08.172820 753 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790722213368738-14927-0/minicluster-data/ts-1/data/instance:
uuid: "071ed94da7b14cb1a61550a7af68ec6f"
format_stamp: "Formatted at 2026-09-29 22:54:08 on dist-test-slave-98mk"
server_key: "7aaeea06d36906d6d05eb64a9580d4de"
server_key_iv: "60722adc26f9df8e80c6ea44f7e3eda6"
server_key_version: "encryptionkey@0"
I20260929 22:54:08.173311 753 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790722213368738-14927-0/minicluster-data/ts-1/wal/instance:
uuid: "071ed94da7b14cb1a61550a7af68ec6f"
format_stamp: "Formatted at 2026-09-29 22:54:08 on dist-test-slave-98mk"
server_key: "7aaeea06d36906d6d05eb64a9580d4de"
server_key_iv: "60722adc26f9df8e80c6ea44f7e3eda6"
server_key_version: "encryptionkey@0"
I20260929 22:54:08.174675 753 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260929 22:54:08.175354 768 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 22:54:08.175513 753 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260929 22:54:08.175598 753 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790722213368738-14927-0/minicluster-data/ts-1/data,/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790722213368738-14927-0/minicluster-data/ts-1/wal
uuid: "071ed94da7b14cb1a61550a7af68ec6f"
format_stamp: "Formatted at 2026-09-29 22:54:08 on dist-test-slave-98mk"
server_key: "7aaeea06d36906d6d05eb64a9580d4de"
server_key_iv: "60722adc26f9df8e80c6ea44f7e3eda6"
server_key_version: "encryptionkey@0"
I20260929 22:54:08.175663 753 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790722213368738-14927-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790722213368738-14927-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790722213368738-14927-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 22:54:08.197734 753 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 22:54:08.199937 753 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 22:54:08.200090 753 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 22:54:08.200310 753 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 22:54:08.200663 753 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 22:54:08.200699 753 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 22:54:08.200729 753 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 22:54:08.200753 753 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 22:54:08.206323 753 rpc_server.cc:307] RPC server started. Bound to: 127.14.147.194:46351
I20260929 22:54:08.206374 881 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.147.194:46351 every 8 connection(s)
I20260929 22:54:08.206693 753 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790722213368738-14927-0/minicluster-data/ts-1/data/info.pb
I20260929 22:54:08.210742 14927 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhWOCLk/build/release/bin/kudu as pid 753
I20260929 22:54:08.210871 14927 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790722213368738-14927-0/minicluster-data/ts-1/wal/instance
I20260929 22:54:08.211043 14927 external_mini_cluster.cc:1428] Setting key 5084c02cf9432cfcfa749c60bfaafef4
Sep 29 22:54:08 dist-test-slave-98mk krb5kdc[511](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790722448, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.147.194@KRBTEST.COM for kudu/127.14.147.254@KRBTEST.COM
I20260929 22:54:08.216706 597 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:54:08.207456 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.147.194:46161 (local address 127.14.147.254:34313)
0929 22:54:08.207505 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0929 22:54:08.207507 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 22:54:08.207699 (+ 192us) server_negotiation.cc:407] Connection header received
0929 22:54:08.208306 (+ 607us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 22:54:08.208309 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 22:54:08.208327 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 22:54:08.208419 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0929 22:54:08.209492 (+ 1073us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:54:08.210010 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:54:08.211118 (+ 1108us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:54:08.211396 (+ 278us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:54:08.211509 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 22:54:08.214065 (+ 2556us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 22:54:08.214066 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 22:54:08.214066 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 22:54:08.214073 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0929 22:54:08.215224 (+ 1151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 22:54:08.215742 (+ 518us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 22:54:08.215742 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 22:54:08.215743 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 22:54:08.215811 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 22:54:08.216082 (+ 271us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 22:54:08.216083 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 22:54:08.216083 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 22:54:08.216186 (+ 103us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 22:54:08.216231 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0929 22:54:08.216571 (+ 340us) server_negotiation.cc:299] Negotiation successful
0929 22:54:08.216638 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260929 22:54:08.216995 884 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:54:08.207392 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.147.254:34313 (local address 127.14.147.194:46161)
0929 22:54:08.207578 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0929 22:54:08.207622 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0929 22:54:08.208107 (+ 485us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 22:54:08.208505 (+ 398us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 22:54:08.208508 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 22:54:08.209101 (+ 593us) client_negotiation.cc:189] Negotiated authn=SASL
0929 22:54:08.209322 (+ 221us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 22:54:08.209327 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 22:54:08.210105 (+ 778us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 22:54:08.210105 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 22:54:08.211023 (+ 918us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 22:54:08.211026 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 22:54:08.211542 (+ 516us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 22:54:08.211543 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 22:54:08.211674 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 22:54:08.212233 (+ 559us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 22:54:08.212243 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0929 22:54:08.213961 (+ 1718us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 22:54:08.215304 (+ 1343us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 22:54:08.215314 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 22:54:08.215316 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 22:54:08.215670 (+ 354us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 22:54:08.215872 (+ 202us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 22:54:08.215873 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 22:54:08.215874 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 22:54:08.216025 (+ 151us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 22:54:08.216241 (+ 216us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 22:54:08.216247 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 22:54:08.216427 (+ 180us) client_negotiation.cc:769] Sending connection context
0929 22:54:08.216563 (+ 136us) client_negotiation.cc:240] Negotiation successful
0929 22:54:08.216746 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":144,"thread_start_us":92,"threads_started":1}
I20260929 22:54:08.217499 882 heartbeater.cc:347] Connected to a master server at 127.14.147.254:34313
I20260929 22:54:08.217592 882 heartbeater.cc:464] Registering TS with master...
I20260929 22:54:08.217763 882 heartbeater.cc:511] Master 127.14.147.254:34313 requested a full tablet report, sending...
I20260929 22:54:08.218118 556 ts_manager.cc:194] Registered new tserver with Master: 071ed94da7b14cb1a61550a7af68ec6f (127.14.147.194:46351)
I20260929 22:54:08.218482 556 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.147.194@KRBTEST.COM'} at 127.14.147.194:46161
WARNING: no policy specified for kudu/127.14.147.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.147.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.147.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790722213368738-14927-0/krb5kdc/kudu_127.14.147.195.keytab.
Entry for principal kudu/127.14.147.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790722213368738-14927-0/krb5kdc/kudu_127.14.147.195.keytab.
I20260929 22:54:08.236004 14927 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhWOCLk/build/release/bin/kudu
/tmp/dist-test-taskhWOCLk/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790722213368738-14927-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790722213368738-14927-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790722213368738-14927-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790722213368738-14927-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.147.195:0
--local_ip_for_outbound_sockets=127.14.147.195
--webserver_interface=127.14.147.195
--webserver_port=0
--tserver_master_addrs=127.14.147.254:34313
--keytab_file=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790722213368738-14927-0/krb5kdc/kudu_127.14.147.195.keytab
--principal=kudu/127.14.147.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.147.212:44843
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790722213368738-14927-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790722213368738-14927-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790722213368738-14927-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 22:54:08.316901 887 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 22:54:08.317129 887 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 22:54:08.317157 887 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 22:54:08.318379 887 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 22:54:08.318432 887 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 22:54:08.318471 887 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.147.195
I20260929 22:54:08.320060 887 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.147.212:44843
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790722213368738-14927-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790722213368738-14927-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790722213368738-14927-0/krb5kdc/kudu_127.14.147.195.keytab
--principal=kudu/127.14.147.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.147.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790722213368738-14927-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.14.147.195
--webserver_port=0
--tserver_master_addrs=127.14.147.254:34313
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.887
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.147.195
--log_dir=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790722213368738-14927-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 21:58:54 UTC on ccycloud.quasar-xguoti.root.comops.site
build id 4681390
I20260929 22:54:08.320310 887 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 22:54:08.320551 887 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260929 22:54:08.325707 887 server_base.cc:1034] running on GCE node
W20260929 22:54:08.325683 893 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 22:54:08.325682 895 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 22:54:08.325683 892 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 22:54:08.326123 887 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 22:54:08.326388 887 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 22:54:08.327528 887 hybrid_clock.cc:648] HybridClock initialized: now 1790722448327514 us; error 30 us; skew 500 ppm
Sep 29 22:54:08 dist-test-slave-98mk krb5kdc[511](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790722448, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.147.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 22:54:08.329926 887 init.cc:377] Logged in from keytab as kudu/127.14.147.195@KRBTEST.COM (short username kudu)
I20260929 22:54:08.330375 887 webserver.cc:460] Webserver started at http://127.14.147.195:38999/ using document root <none> and password file <none>
I20260929 22:54:08.330588 887 fs_manager.cc:359] Metadata directory not provided
I20260929 22:54:08.330631 887 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 22:54:08.330749 887 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 22:54:08.331854 887 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790722213368738-14927-0/minicluster-data/ts-2/data/instance:
uuid: "ab311df44dad4bbaba2456869b936cda"
format_stamp: "Formatted at 2026-09-29 22:54:08 on dist-test-slave-98mk"
server_key: "248b806d376f72ee4201b10210e3f7fb"
server_key_iv: "bd7142f981b37447d241924294642108"
server_key_version: "encryptionkey@0"
I20260929 22:54:08.332177 887 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790722213368738-14927-0/minicluster-data/ts-2/wal/instance:
uuid: "ab311df44dad4bbaba2456869b936cda"
format_stamp: "Formatted at 2026-09-29 22:54:08 on dist-test-slave-98mk"
server_key: "248b806d376f72ee4201b10210e3f7fb"
server_key_iv: "bd7142f981b37447d241924294642108"
server_key_version: "encryptionkey@0"
I20260929 22:54:08.333669 887 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260929 22:54:08.334462 902 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 22:54:08.334656 887 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260929 22:54:08.334748 887 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790722213368738-14927-0/minicluster-data/ts-2/data,/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790722213368738-14927-0/minicluster-data/ts-2/wal
uuid: "ab311df44dad4bbaba2456869b936cda"
format_stamp: "Formatted at 2026-09-29 22:54:08 on dist-test-slave-98mk"
server_key: "248b806d376f72ee4201b10210e3f7fb"
server_key_iv: "bd7142f981b37447d241924294642108"
server_key_version: "encryptionkey@0"
I20260929 22:54:08.334807 887 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790722213368738-14927-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790722213368738-14927-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790722213368738-14927-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 22:54:08.346827 887 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 22:54:08.348832 887 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 22:54:08.348955 887 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 22:54:08.349169 887 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 22:54:08.349439 887 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 22:54:08.349469 887 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 22:54:08.349493 887 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 22:54:08.349509 887 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 22:54:08.355382 887 rpc_server.cc:307] RPC server started. Bound to: 127.14.147.195:33065
I20260929 22:54:08.355476 1015 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.147.195:33065 every 8 connection(s)
I20260929 22:54:08.355803 887 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790722213368738-14927-0/minicluster-data/ts-2/data/info.pb
I20260929 22:54:08.361068 14927 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhWOCLk/build/release/bin/kudu as pid 887
I20260929 22:54:08.361169 14927 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790722213368738-14927-0/minicluster-data/ts-2/wal/instance
I20260929 22:54:08.361351 14927 external_mini_cluster.cc:1428] Setting key 0ea1aa471d4558c4682b9b283ac9ddd1
Sep 29 22:54:08 dist-test-slave-98mk krb5kdc[511](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790722448, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.147.195@KRBTEST.COM for kudu/127.14.147.254@KRBTEST.COM
I20260929 22:54:08.365569 1018 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:54:08.356520 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.147.254:34313 (local address 127.14.147.195:58497)
0929 22:54:08.356667 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0929 22:54:08.356704 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0929 22:54:08.357165 (+ 461us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 22:54:08.357362 (+ 197us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 22:54:08.357365 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 22:54:08.357927 (+ 562us) client_negotiation.cc:189] Negotiated authn=SASL
0929 22:54:08.358123 (+ 196us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 22:54:08.358128 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 22:54:08.358904 (+ 776us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 22:54:08.358906 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 22:54:08.360175 (+ 1269us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 22:54:08.360181 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 22:54:08.360624 (+ 443us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 22:54:08.360625 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 22:54:08.360764 (+ 139us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 22:54:08.361408 (+ 644us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 22:54:08.361421 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0929 22:54:08.362928 (+ 1507us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 22:54:08.364284 (+ 1356us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 22:54:08.364291 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 22:54:08.364293 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 22:54:08.364536 (+ 243us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 22:54:08.364684 (+ 148us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 22:54:08.364685 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 22:54:08.364685 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 22:54:08.364779 (+ 94us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 22:54:08.364993 (+ 214us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 22:54:08.364999 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 22:54:08.365143 (+ 144us) client_negotiation.cc:769] Sending connection context
0929 22:54:08.365236 (+ 93us) client_negotiation.cc:240] Negotiation successful
0929 22:54:08.365376 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":69,"threads_started":1}
I20260929 22:54:08.365576 597 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:54:08.356760 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.147.195:58497 (local address 127.14.147.254:34313)
0929 22:54:08.356801 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0929 22:54:08.356803 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 22:54:08.356813 (+ 10us) server_negotiation.cc:407] Connection header received
0929 22:54:08.357247 (+ 434us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 22:54:08.357248 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 22:54:08.357257 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 22:54:08.357303 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0929 22:54:08.358218 (+ 915us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:54:08.358728 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:54:08.360300 (+ 1572us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:54:08.360515 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:54:08.360576 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 22:54:08.363011 (+ 2435us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 22:54:08.363012 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 22:54:08.363013 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 22:54:08.363020 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0929 22:54:08.364201 (+ 1181us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 22:54:08.364595 (+ 394us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 22:54:08.364596 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 22:54:08.364596 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 22:54:08.364641 (+ 45us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 22:54:08.364820 (+ 179us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 22:54:08.364820 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 22:54:08.364821 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 22:54:08.364925 (+ 104us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 22:54:08.364971 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0929 22:54:08.365353 (+ 382us) server_negotiation.cc:299] Negotiation successful
0929 22:54:08.365484 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260929 22:54:08.366149 1016 heartbeater.cc:347] Connected to a master server at 127.14.147.254:34313
I20260929 22:54:08.366256 1016 heartbeater.cc:464] Registering TS with master...
I20260929 22:54:08.366415 1016 heartbeater.cc:511] Master 127.14.147.254:34313 requested a full tablet report, sending...
I20260929 22:54:08.366799 556 ts_manager.cc:194] Registered new tserver with Master: ab311df44dad4bbaba2456869b936cda (127.14.147.195:33065)
I20260929 22:54:08.367168 556 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.147.195@KRBTEST.COM'} at 127.14.147.195:58497
I20260929 22:54:08.373528 14927 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 22:54:08.381026 597 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:54:08.374899 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45534 (local address 127.14.147.254:34313)
0929 22:54:08.374957 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0929 22:54:08.374960 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 22:54:08.375061 (+ 101us) server_negotiation.cc:407] Connection header received
0929 22:54:08.375109 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 22:54:08.375109 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 22:54:08.375118 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 22:54:08.375165 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0929 22:54:08.375736 (+ 571us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:54:08.376238 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:54:08.377016 (+ 778us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:54:08.377208 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:54:08.377269 (+ 61us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 22:54:08.378612 (+ 1343us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 22:54:08.378619 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 22:54:08.378619 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 22:54:08.378626 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0929 22:54:08.379833 (+ 1207us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 22:54:08.380196 (+ 363us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 22:54:08.380197 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 22:54:08.380197 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 22:54:08.380254 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 22:54:08.380413 (+ 159us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 22:54:08.380414 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 22:54:08.380414 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 22:54:08.380547 (+ 133us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 22:54:08.380630 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0929 22:54:08.380832 (+ 202us) server_negotiation.cc:299] Negotiation successful
0929 22:54:08.380923 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260929 22:54:08.382813 556 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45534:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 22:54:08.384686 556 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 22:54:08.390709 1025 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:54:08.386709 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.147.195:33065 (local address 127.0.0.1:40272)
0929 22:54:08.387074 (+ 365us) negotiation.cc:107] Waiting for socket to connect
0929 22:54:08.387082 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0929 22:54:08.387116 (+ 34us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 22:54:08.387310 (+ 194us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 22:54:08.387313 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 22:54:08.387315 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 22:54:08.387399 (+ 84us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 22:54:08.387401 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 22:54:08.388443 (+ 1042us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 22:54:08.388443 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 22:54:08.389485 (+ 1042us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 22:54:08.389488 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 22:54:08.390393 (+ 905us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 22:54:08.390394 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 22:54:08.390475 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 22:54:08.390490 (+ 15us) client_negotiation.cc:769] Sending connection context
0929 22:54:08.390563 (+ 73us) client_negotiation.cc:240] Negotiation successful
0929 22:54:08.390619 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":354,"thread_start_us":65,"threads_started":1}
I20260929 22:54:08.390718 1028 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:54:08.386971 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40272 (local address 127.14.147.195:33065)
0929 22:54:08.387159 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0929 22:54:08.387163 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 22:54:08.387173 (+ 10us) server_negotiation.cc:407] Connection header received
0929 22:54:08.387213 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 22:54:08.387215 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 22:54:08.387232 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 22:54:08.387302 (+ 70us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 22:54:08.387511 (+ 209us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:54:08.388344 (+ 833us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:54:08.389616 (+ 1272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:54:08.390315 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:54:08.390391 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 22:54:08.390458 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0929 22:54:08.390584 (+ 126us) server_negotiation.cc:299] Negotiation successful
0929 22:54:08.390633 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":158,"thread_start_us":81,"threads_started":1}
I20260929 22:54:08.390937 1024 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:54:08.386708 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.147.193:42539 (local address 127.0.0.1:41820)
0929 22:54:08.386880 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0929 22:54:08.386911 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0929 22:54:08.386987 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 22:54:08.387503 (+ 516us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 22:54:08.387504 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 22:54:08.387506 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 22:54:08.387569 (+ 63us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 22:54:08.387571 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 22:54:08.388884 (+ 1313us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 22:54:08.388885 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 22:54:08.389785 (+ 900us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 22:54:08.389787 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 22:54:08.390698 (+ 911us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 22:54:08.390699 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 22:54:08.390761 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 22:54:08.390765 (+ 4us) client_negotiation.cc:769] Sending connection context
0929 22:54:08.390810 (+ 45us) client_negotiation.cc:240] Negotiation successful
0929 22:54:08.390852 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":150,"mutex_wait_us":38,"thread_start_us":70,"threads_started":1}
I20260929 22:54:08.391156 1026 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:54:08.386708 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.147.194:46351 (local address 127.0.0.1:43622)
0929 22:54:08.387170 (+ 462us) negotiation.cc:107] Waiting for socket to connect
0929 22:54:08.387178 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0929 22:54:08.387221 (+ 43us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 22:54:08.387496 (+ 275us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 22:54:08.387498 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 22:54:08.387500 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 22:54:08.387717 (+ 217us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 22:54:08.387719 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 22:54:08.388978 (+ 1259us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 22:54:08.388979 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 22:54:08.389970 (+ 991us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 22:54:08.389972 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 22:54:08.390963 (+ 991us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 22:54:08.390964 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 22:54:08.391022 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 22:54:08.391027 (+ 5us) client_negotiation.cc:769] Sending connection context
0929 22:54:08.391077 (+ 50us) client_negotiation.cc:240] Negotiation successful
0929 22:54:08.391109 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":454,"mutex_wait_us":23,"thread_start_us":70,"threads_started":1}
I20260929 22:54:08.391249 1029 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:54:08.387164 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41820 (local address 127.14.147.193:42539)
0929 22:54:08.387345 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0929 22:54:08.387349 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 22:54:08.387358 (+ 9us) server_negotiation.cc:407] Connection header received
0929 22:54:08.387391 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 22:54:08.387393 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 22:54:08.387406 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 22:54:08.387504 (+ 98us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 22:54:08.387645 (+ 141us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:54:08.388771 (+ 1126us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:54:08.389878 (+ 1107us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:54:08.390554 (+ 676us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:54:08.390616 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 22:54:08.390665 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0929 22:54:08.391103 (+ 438us) server_negotiation.cc:299] Negotiation successful
0929 22:54:08.391148 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":154,"thread_start_us":79,"threads_started":1}
I20260929 22:54:08.391405 1027 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:54:08.386955 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43622 (local address 127.14.147.194:46351)
0929 22:54:08.387128 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0929 22:54:08.387133 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 22:54:08.387230 (+ 97us) server_negotiation.cc:407] Connection header received
0929 22:54:08.387278 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 22:54:08.387281 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 22:54:08.387296 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 22:54:08.387382 (+ 86us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 22:54:08.387793 (+ 411us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:54:08.388881 (+ 1088us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:54:08.390045 (+ 1164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:54:08.390612 (+ 567us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:54:08.390869 (+ 257us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 22:54:08.390934 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0929 22:54:08.391279 (+ 345us) server_negotiation.cc:299] Negotiation successful
0929 22:54:08.391332 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":137,"thread_start_us":88,"threads_started":1}
I20260929 22:54:08.391865 682 tablet_service.cc:1467] Processing CreateTablet for tablet 49c49116363b4a61846bdd1c724737da (DEFAULT_TABLE table=test-table [id=15d87b0dbb1d442e9d2636c414a0b3fb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 22:54:08.391851 950 tablet_service.cc:1467] Processing CreateTablet for tablet 49c49116363b4a61846bdd1c724737da (DEFAULT_TABLE table=test-table [id=15d87b0dbb1d442e9d2636c414a0b3fb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 22:54:08.392015 816 tablet_service.cc:1467] Processing CreateTablet for tablet 49c49116363b4a61846bdd1c724737da (DEFAULT_TABLE table=test-table [id=15d87b0dbb1d442e9d2636c414a0b3fb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 22:54:08.392226 682 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 49c49116363b4a61846bdd1c724737da. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 22:54:08.392228 950 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 49c49116363b4a61846bdd1c724737da. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 22:54:08.392269 816 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 49c49116363b4a61846bdd1c724737da. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 22:54:08.394862 1030 tablet_bootstrap.cc:492] T 49c49116363b4a61846bdd1c724737da P ab311df44dad4bbaba2456869b936cda: Bootstrap starting.
I20260929 22:54:08.395607 1030 tablet_bootstrap.cc:654] T 49c49116363b4a61846bdd1c724737da P ab311df44dad4bbaba2456869b936cda: Neither blocks nor log segments found. Creating new log.
I20260929 22:54:08.395759 1032 tablet_bootstrap.cc:492] T 49c49116363b4a61846bdd1c724737da P 5ca5d8fcb5ea447ebc9554162325cc77: Bootstrap starting.
I20260929 22:54:08.395906 1031 tablet_bootstrap.cc:492] T 49c49116363b4a61846bdd1c724737da P 071ed94da7b14cb1a61550a7af68ec6f: Bootstrap starting.
I20260929 22:54:08.396147 1030 log.cc:824] T 49c49116363b4a61846bdd1c724737da P ab311df44dad4bbaba2456869b936cda: Log is configured to *not* fsync() on all Append() calls
I20260929 22:54:08.396368 1032 tablet_bootstrap.cc:654] T 49c49116363b4a61846bdd1c724737da P 5ca5d8fcb5ea447ebc9554162325cc77: Neither blocks nor log segments found. Creating new log.
I20260929 22:54:08.396484 1031 tablet_bootstrap.cc:654] T 49c49116363b4a61846bdd1c724737da P 071ed94da7b14cb1a61550a7af68ec6f: Neither blocks nor log segments found. Creating new log.
I20260929 22:54:08.396867 1032 log.cc:824] T 49c49116363b4a61846bdd1c724737da P 5ca5d8fcb5ea447ebc9554162325cc77: Log is configured to *not* fsync() on all Append() calls
I20260929 22:54:08.396916 1031 log.cc:824] T 49c49116363b4a61846bdd1c724737da P 071ed94da7b14cb1a61550a7af68ec6f: Log is configured to *not* fsync() on all Append() calls
I20260929 22:54:08.396929 1030 tablet_bootstrap.cc:492] T 49c49116363b4a61846bdd1c724737da P ab311df44dad4bbaba2456869b936cda: No bootstrap required, opened a new log
I20260929 22:54:08.397065 1030 ts_tablet_manager.cc:1397] T 49c49116363b4a61846bdd1c724737da P ab311df44dad4bbaba2456869b936cda: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260929 22:54:08.397563 1031 tablet_bootstrap.cc:492] T 49c49116363b4a61846bdd1c724737da P 071ed94da7b14cb1a61550a7af68ec6f: No bootstrap required, opened a new log
I20260929 22:54:08.397617 1031 ts_tablet_manager.cc:1397] T 49c49116363b4a61846bdd1c724737da P 071ed94da7b14cb1a61550a7af68ec6f: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260929 22:54:08.397773 1032 tablet_bootstrap.cc:492] T 49c49116363b4a61846bdd1c724737da P 5ca5d8fcb5ea447ebc9554162325cc77: No bootstrap required, opened a new log
I20260929 22:54:08.397841 1032 ts_tablet_manager.cc:1397] T 49c49116363b4a61846bdd1c724737da P 5ca5d8fcb5ea447ebc9554162325cc77: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260929 22:54:08.399199 1031 raft_consensus.cc:348] T 49c49116363b4a61846bdd1c724737da P 071ed94da7b14cb1a61550a7af68ec6f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "071ed94da7b14cb1a61550a7af68ec6f" member_type: VOTER last_known_addr { host: "127.14.147.194" port: 46351 } } peers { permanent_uuid: "5ca5d8fcb5ea447ebc9554162325cc77" member_type: VOTER last_known_addr { host: "127.14.147.193" port: 42539 } } peers { permanent_uuid: "ab311df44dad4bbaba2456869b936cda" member_type: VOTER last_known_addr { host: "127.14.147.195" port: 33065 } }
I20260929 22:54:08.399156 1030 raft_consensus.cc:348] T 49c49116363b4a61846bdd1c724737da P ab311df44dad4bbaba2456869b936cda [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "071ed94da7b14cb1a61550a7af68ec6f" member_type: VOTER last_known_addr { host: "127.14.147.194" port: 46351 } } peers { permanent_uuid: "5ca5d8fcb5ea447ebc9554162325cc77" member_type: VOTER last_known_addr { host: "127.14.147.193" port: 42539 } } peers { permanent_uuid: "ab311df44dad4bbaba2456869b936cda" member_type: VOTER last_known_addr { host: "127.14.147.195" port: 33065 } }
I20260929 22:54:08.399341 1031 raft_consensus.cc:374] T 49c49116363b4a61846bdd1c724737da P 071ed94da7b14cb1a61550a7af68ec6f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 22:54:08.399364 1031 raft_consensus.cc:729] T 49c49116363b4a61846bdd1c724737da P 071ed94da7b14cb1a61550a7af68ec6f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 071ed94da7b14cb1a61550a7af68ec6f, State: Initialized, Role: FOLLOWER
I20260929 22:54:08.399442 1030 raft_consensus.cc:374] T 49c49116363b4a61846bdd1c724737da P ab311df44dad4bbaba2456869b936cda [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 22:54:08.399513 1030 raft_consensus.cc:729] T 49c49116363b4a61846bdd1c724737da P ab311df44dad4bbaba2456869b936cda [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ab311df44dad4bbaba2456869b936cda, State: Initialized, Role: FOLLOWER
I20260929 22:54:08.399539 1031 consensus_queue.cc:260] T 49c49116363b4a61846bdd1c724737da P 071ed94da7b14cb1a61550a7af68ec6f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "071ed94da7b14cb1a61550a7af68ec6f" member_type: VOTER last_known_addr { host: "127.14.147.194" port: 46351 } } peers { permanent_uuid: "5ca5d8fcb5ea447ebc9554162325cc77" member_type: VOTER last_known_addr { host: "127.14.147.193" port: 42539 } } peers { permanent_uuid: "ab311df44dad4bbaba2456869b936cda" member_type: VOTER last_known_addr { host: "127.14.147.195" port: 33065 } }
I20260929 22:54:08.399765 1030 consensus_queue.cc:260] T 49c49116363b4a61846bdd1c724737da P ab311df44dad4bbaba2456869b936cda [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "071ed94da7b14cb1a61550a7af68ec6f" member_type: VOTER last_known_addr { host: "127.14.147.194" port: 46351 } } peers { permanent_uuid: "5ca5d8fcb5ea447ebc9554162325cc77" member_type: VOTER last_known_addr { host: "127.14.147.193" port: 42539 } } peers { permanent_uuid: "ab311df44dad4bbaba2456869b936cda" member_type: VOTER last_known_addr { host: "127.14.147.195" port: 33065 } }
I20260929 22:54:08.399850 882 heartbeater.cc:503] Master 127.14.147.254:34313 was elected leader, sending a full tablet report...
I20260929 22:54:08.400013 1032 raft_consensus.cc:348] T 49c49116363b4a61846bdd1c724737da P 5ca5d8fcb5ea447ebc9554162325cc77 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "071ed94da7b14cb1a61550a7af68ec6f" member_type: VOTER last_known_addr { host: "127.14.147.194" port: 46351 } } peers { permanent_uuid: "5ca5d8fcb5ea447ebc9554162325cc77" member_type: VOTER last_known_addr { host: "127.14.147.193" port: 42539 } } peers { permanent_uuid: "ab311df44dad4bbaba2456869b936cda" member_type: VOTER last_known_addr { host: "127.14.147.195" port: 33065 } }
I20260929 22:54:08.400148 1016 heartbeater.cc:503] Master 127.14.147.254:34313 was elected leader, sending a full tablet report...
I20260929 22:54:08.400175 1032 raft_consensus.cc:374] T 49c49116363b4a61846bdd1c724737da P 5ca5d8fcb5ea447ebc9554162325cc77 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 22:54:08.400208 1032 raft_consensus.cc:729] T 49c49116363b4a61846bdd1c724737da P 5ca5d8fcb5ea447ebc9554162325cc77 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5ca5d8fcb5ea447ebc9554162325cc77, State: Initialized, Role: FOLLOWER
I20260929 22:54:08.399847 1031 ts_tablet_manager.cc:1428] T 49c49116363b4a61846bdd1c724737da P 071ed94da7b14cb1a61550a7af68ec6f: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260929 22:54:08.400331 1032 consensus_queue.cc:260] T 49c49116363b4a61846bdd1c724737da P 5ca5d8fcb5ea447ebc9554162325cc77 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "071ed94da7b14cb1a61550a7af68ec6f" member_type: VOTER last_known_addr { host: "127.14.147.194" port: 46351 } } peers { permanent_uuid: "5ca5d8fcb5ea447ebc9554162325cc77" member_type: VOTER last_known_addr { host: "127.14.147.193" port: 42539 } } peers { permanent_uuid: "ab311df44dad4bbaba2456869b936cda" member_type: VOTER last_known_addr { host: "127.14.147.195" port: 33065 } }
I20260929 22:54:08.400141 1030 ts_tablet_manager.cc:1428] T 49c49116363b4a61846bdd1c724737da P ab311df44dad4bbaba2456869b936cda: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260929 22:54:08.400604 1032 ts_tablet_manager.cc:1428] T 49c49116363b4a61846bdd1c724737da P 5ca5d8fcb5ea447ebc9554162325cc77: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260929 22:54:08.400719 748 heartbeater.cc:503] Master 127.14.147.254:34313 was elected leader, sending a full tablet report...
I20260929 22:54:08.440873 1038 raft_consensus.cc:482] T 49c49116363b4a61846bdd1c724737da P 5ca5d8fcb5ea447ebc9554162325cc77 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 22:54:08.440992 1038 raft_consensus.cc:504] T 49c49116363b4a61846bdd1c724737da P 5ca5d8fcb5ea447ebc9554162325cc77 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "071ed94da7b14cb1a61550a7af68ec6f" member_type: VOTER last_known_addr { host: "127.14.147.194" port: 46351 } } peers { permanent_uuid: "5ca5d8fcb5ea447ebc9554162325cc77" member_type: VOTER last_known_addr { host: "127.14.147.193" port: 42539 } } peers { permanent_uuid: "ab311df44dad4bbaba2456869b936cda" member_type: VOTER last_known_addr { host: "127.14.147.195" port: 33065 } }
I20260929 22:54:08.441372 1038 leader_election.cc:288] T 49c49116363b4a61846bdd1c724737da P 5ca5d8fcb5ea447ebc9554162325cc77 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 071ed94da7b14cb1a61550a7af68ec6f (127.14.147.194:46351), ab311df44dad4bbaba2456869b936cda (127.14.147.195:33065)
I20260929 22:54:08.444118 1028 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:54:08.441648 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.147.193:33425 (local address 127.14.147.195:33065)
0929 22:54:08.441686 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0929 22:54:08.441688 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 22:54:08.441815 (+ 127us) server_negotiation.cc:407] Connection header received
0929 22:54:08.441853 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 22:54:08.441854 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 22:54:08.441863 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 22:54:08.441906 (+ 43us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 22:54:08.442039 (+ 133us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:54:08.442492 (+ 453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:54:08.443446 (+ 954us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:54:08.443806 (+ 360us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:54:08.443876 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 22:54:08.443886 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0929 22:54:08.444016 (+ 130us) server_negotiation.cc:299] Negotiation successful
0929 22:54:08.444049 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260929 22:54:08.444155 750 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:54:08.441548 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.147.194:46351 (local address 127.14.147.193:60059)
0929 22:54:08.441636 (+ 88us) negotiation.cc:107] Waiting for socket to connect
0929 22:54:08.441647 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0929 22:54:08.441697 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 22:54:08.441838 (+ 141us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 22:54:08.441839 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 22:54:08.441841 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 22:54:08.441921 (+ 80us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 22:54:08.441923 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 22:54:08.442535 (+ 612us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 22:54:08.442536 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 22:54:08.443336 (+ 800us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 22:54:08.443339 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 22:54:08.443907 (+ 568us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 22:54:08.443908 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 22:54:08.443979 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 22:54:08.443981 (+ 2us) client_negotiation.cc:769] Sending connection context
0929 22:54:08.444026 (+ 45us) client_negotiation.cc:240] Negotiation successful
0929 22:54:08.444062 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":67,"mutex_wait_us":45}
I20260929 22:54:08.444144 1027 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:54:08.441636 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.147.193:60059 (local address 127.14.147.194:46351)
0929 22:54:08.441690 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0929 22:54:08.441693 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 22:54:08.441703 (+ 10us) server_negotiation.cc:407] Connection header received
0929 22:54:08.441757 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 22:54:08.441758 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 22:54:08.441768 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 22:54:08.441831 (+ 63us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 22:54:08.441983 (+ 152us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:54:08.442466 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:54:08.443468 (+ 1002us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:54:08.443839 (+ 371us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:54:08.443900 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 22:54:08.443909 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0929 22:54:08.444040 (+ 131us) server_negotiation.cc:299] Negotiation successful
0929 22:54:08.444075 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260929 22:54:08.444169 1039 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:54:08.441551 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.147.195:33065 (local address 127.14.147.193:33425)
0929 22:54:08.441747 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0929 22:54:08.441756 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0929 22:54:08.441805 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 22:54:08.441916 (+ 111us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 22:54:08.441917 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 22:54:08.441918 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 22:54:08.441983 (+ 65us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 22:54:08.441985 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 22:54:08.442562 (+ 577us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 22:54:08.442563 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 22:54:08.443335 (+ 772us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 22:54:08.443337 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 22:54:08.443894 (+ 557us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 22:54:08.443895 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 22:54:08.443957 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 22:54:08.443960 (+ 3us) client_negotiation.cc:769] Sending connection context
0929 22:54:08.444004 (+ 44us) client_negotiation.cc:240] Negotiation successful
0929 22:54:08.444043 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":182,"mutex_wait_us":14,"thread_start_us":107,"threads_started":1}
I20260929 22:54:08.444499 836 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "49c49116363b4a61846bdd1c724737da" candidate_uuid: "5ca5d8fcb5ea447ebc9554162325cc77" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "071ed94da7b14cb1a61550a7af68ec6f" is_pre_election: true
I20260929 22:54:08.444528 970 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "49c49116363b4a61846bdd1c724737da" candidate_uuid: "5ca5d8fcb5ea447ebc9554162325cc77" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ab311df44dad4bbaba2456869b936cda" is_pre_election: true
I20260929 22:54:08.444643 836 raft_consensus.cc:2454] T 49c49116363b4a61846bdd1c724737da P 071ed94da7b14cb1a61550a7af68ec6f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5ca5d8fcb5ea447ebc9554162325cc77 in term 0.
I20260929 22:54:08.444681 970 raft_consensus.cc:2454] T 49c49116363b4a61846bdd1c724737da P ab311df44dad4bbaba2456869b936cda [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5ca5d8fcb5ea447ebc9554162325cc77 in term 0.
I20260929 22:54:08.444883 635 leader_election.cc:302] T 49c49116363b4a61846bdd1c724737da P 5ca5d8fcb5ea447ebc9554162325cc77 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5ca5d8fcb5ea447ebc9554162325cc77, ab311df44dad4bbaba2456869b936cda; no voters:
I20260929 22:54:08.445017 1038 raft_consensus.cc:2781] T 49c49116363b4a61846bdd1c724737da P 5ca5d8fcb5ea447ebc9554162325cc77 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 22:54:08.445089 1038 raft_consensus.cc:482] T 49c49116363b4a61846bdd1c724737da P 5ca5d8fcb5ea447ebc9554162325cc77 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 22:54:08.445114 1038 raft_consensus.cc:3037] T 49c49116363b4a61846bdd1c724737da P 5ca5d8fcb5ea447ebc9554162325cc77 [term 0 FOLLOWER]: Advancing to term 1
I20260929 22:54:08.445935 1038 raft_consensus.cc:504] T 49c49116363b4a61846bdd1c724737da P 5ca5d8fcb5ea447ebc9554162325cc77 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "071ed94da7b14cb1a61550a7af68ec6f" member_type: VOTER last_known_addr { host: "127.14.147.194" port: 46351 } } peers { permanent_uuid: "5ca5d8fcb5ea447ebc9554162325cc77" member_type: VOTER last_known_addr { host: "127.14.147.193" port: 42539 } } peers { permanent_uuid: "ab311df44dad4bbaba2456869b936cda" member_type: VOTER last_known_addr { host: "127.14.147.195" port: 33065 } }
I20260929 22:54:08.446056 1038 leader_election.cc:288] T 49c49116363b4a61846bdd1c724737da P 5ca5d8fcb5ea447ebc9554162325cc77 [CANDIDATE]: Term 1 election: Requested vote from peers 071ed94da7b14cb1a61550a7af68ec6f (127.14.147.194:46351), ab311df44dad4bbaba2456869b936cda (127.14.147.195:33065)
I20260929 22:54:08.446241 970 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "49c49116363b4a61846bdd1c724737da" candidate_uuid: "5ca5d8fcb5ea447ebc9554162325cc77" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ab311df44dad4bbaba2456869b936cda"
I20260929 22:54:08.446285 836 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "49c49116363b4a61846bdd1c724737da" candidate_uuid: "5ca5d8fcb5ea447ebc9554162325cc77" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "071ed94da7b14cb1a61550a7af68ec6f"
I20260929 22:54:08.446321 970 raft_consensus.cc:3037] T 49c49116363b4a61846bdd1c724737da P ab311df44dad4bbaba2456869b936cda [term 0 FOLLOWER]: Advancing to term 1
I20260929 22:54:08.446358 836 raft_consensus.cc:3037] T 49c49116363b4a61846bdd1c724737da P 071ed94da7b14cb1a61550a7af68ec6f [term 0 FOLLOWER]: Advancing to term 1
I20260929 22:54:08.447058 970 raft_consensus.cc:2454] T 49c49116363b4a61846bdd1c724737da P ab311df44dad4bbaba2456869b936cda [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5ca5d8fcb5ea447ebc9554162325cc77 in term 1.
I20260929 22:54:08.447062 836 raft_consensus.cc:2454] T 49c49116363b4a61846bdd1c724737da P 071ed94da7b14cb1a61550a7af68ec6f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5ca5d8fcb5ea447ebc9554162325cc77 in term 1.
I20260929 22:54:08.447225 635 leader_election.cc:302] T 49c49116363b4a61846bdd1c724737da P 5ca5d8fcb5ea447ebc9554162325cc77 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5ca5d8fcb5ea447ebc9554162325cc77, ab311df44dad4bbaba2456869b936cda; no voters:
I20260929 22:54:08.447322 1038 raft_consensus.cc:2781] T 49c49116363b4a61846bdd1c724737da P 5ca5d8fcb5ea447ebc9554162325cc77 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 22:54:08.447506 1038 raft_consensus.cc:686] T 49c49116363b4a61846bdd1c724737da P 5ca5d8fcb5ea447ebc9554162325cc77 [term 1 LEADER]: Becoming Leader. State: Replica: 5ca5d8fcb5ea447ebc9554162325cc77, State: Running, Role: LEADER
I20260929 22:54:08.447660 1038 consensus_queue.cc:237] T 49c49116363b4a61846bdd1c724737da P 5ca5d8fcb5ea447ebc9554162325cc77 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "071ed94da7b14cb1a61550a7af68ec6f" member_type: VOTER last_known_addr { host: "127.14.147.194" port: 46351 } } peers { permanent_uuid: "5ca5d8fcb5ea447ebc9554162325cc77" member_type: VOTER last_known_addr { host: "127.14.147.193" port: 42539 } } peers { permanent_uuid: "ab311df44dad4bbaba2456869b936cda" member_type: VOTER last_known_addr { host: "127.14.147.195" port: 33065 } }
I20260929 22:54:08.448478 554 catalog_manager.cc:5495] T 49c49116363b4a61846bdd1c724737da P 5ca5d8fcb5ea447ebc9554162325cc77 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5ca5d8fcb5ea447ebc9554162325cc77 (127.14.147.193). New cstate: current_term: 1 leader_uuid: "5ca5d8fcb5ea447ebc9554162325cc77" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "071ed94da7b14cb1a61550a7af68ec6f" member_type: VOTER last_known_addr { host: "127.14.147.194" port: 46351 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5ca5d8fcb5ea447ebc9554162325cc77" member_type: VOTER last_known_addr { host: "127.14.147.193" port: 42539 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ab311df44dad4bbaba2456869b936cda" member_type: VOTER last_known_addr { host: "127.14.147.195" port: 33065 } health_report { overall_health: UNKNOWN } } }
W20260929 22:54:08.457301 883 tablet.cc:2369] T 49c49116363b4a61846bdd1c724737da P 071ed94da7b14cb1a61550a7af68ec6f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 22:54:08.465003 1029 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:54:08.462931 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41824 (local address 127.14.147.193:42539)
0929 22:54:08.462980 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0929 22:54:08.462982 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 22:54:08.462989 (+ 7us) server_negotiation.cc:407] Connection header received
0929 22:54:08.463024 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 22:54:08.463024 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 22:54:08.463047 (+ 23us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 22:54:08.463110 (+ 63us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 22:54:08.463213 (+ 103us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:54:08.463715 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:54:08.464397 (+ 682us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:54:08.464585 (+ 188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:54:08.464626 (+ 41us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 22:54:08.464728 (+ 102us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 22:54:08.464816 (+ 88us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 22:54:08.464889 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0929 22:54:08.464909 (+ 20us) server_negotiation.cc:299] Negotiation successful
0929 22:54:08.464919 (+ 10us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260929 22:54:08.466912 836 raft_consensus.cc:1261] T 49c49116363b4a61846bdd1c724737da P 071ed94da7b14cb1a61550a7af68ec6f [term 1 FOLLOWER]: Refusing update from remote peer 5ca5d8fcb5ea447ebc9554162325cc77: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 22:54:08.466912 970 raft_consensus.cc:1261] T 49c49116363b4a61846bdd1c724737da P ab311df44dad4bbaba2456869b936cda [term 1 FOLLOWER]: Refusing update from remote peer 5ca5d8fcb5ea447ebc9554162325cc77: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 22:54:08.467239 1038 consensus_queue.cc:1035] T 49c49116363b4a61846bdd1c724737da P 5ca5d8fcb5ea447ebc9554162325cc77 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab311df44dad4bbaba2456869b936cda" member_type: VOTER last_known_addr { host: "127.14.147.195" port: 33065 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 22:54:08.467393 1040 consensus_queue.cc:1035] T 49c49116363b4a61846bdd1c724737da P 5ca5d8fcb5ea447ebc9554162325cc77 [LEADER]: Connected to new peer: Peer: permanent_uuid: "071ed94da7b14cb1a61550a7af68ec6f" member_type: VOTER last_known_addr { host: "127.14.147.194" port: 46351 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 22:54:08.469854 1043 mvcc.cc:205] Tried to move back new op lower bound from 7334799148917342208 to 7334799148842270720. Current Snapshot: MvccSnapshot[applied={T|T < 7334799148917342208}]
I20260929 22:54:08.470729 1044 mvcc.cc:205] Tried to move back new op lower bound from 7334799148917342208 to 7334799148842270720. Current Snapshot: MvccSnapshot[applied={T|T < 7334799148917342208 or (T in {7334799148917342208})}]
I20260929 22:54:08.472378 554 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45534:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 22:54:08.472465 554 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45534:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 22:54:08.473134 554 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 750ef0065148452b93d625c887ae243a: Sending DeleteTablet for 3 replicas of tablet 49c49116363b4a61846bdd1c724737da
I20260929 22:54:08.473497 682 tablet_service.cc:1514] Processing DeleteTablet for tablet 49c49116363b4a61846bdd1c724737da with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 22:54:08 UTC) from {username='kudu', principal='kudu/127.14.147.254@KRBTEST.COM'} at 127.0.0.1:41820
I20260929 22:54:08.473510 816 tablet_service.cc:1514] Processing DeleteTablet for tablet 49c49116363b4a61846bdd1c724737da with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 22:54:08 UTC) from {username='kudu', principal='kudu/127.14.147.254@KRBTEST.COM'} at 127.0.0.1:43622
I20260929 22:54:08.473740 950 tablet_service.cc:1514] Processing DeleteTablet for tablet 49c49116363b4a61846bdd1c724737da with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 22:54:08 UTC) from {username='kudu', principal='kudu/127.14.147.254@KRBTEST.COM'} at 127.0.0.1:40272
I20260929 22:54:08.473945 1054 tablet_replica.cc:330] T 49c49116363b4a61846bdd1c724737da P ab311df44dad4bbaba2456869b936cda: stopping tablet replica
I20260929 22:54:08.474074 1054 raft_consensus.cc:2229] T 49c49116363b4a61846bdd1c724737da P ab311df44dad4bbaba2456869b936cda [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 22:54:08.474197 14927 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhWOCLk/build/release/bin/kudu with pid 619
I20260929 22:54:08.474720 1054 raft_consensus.cc:2258] T 49c49116363b4a61846bdd1c724737da P ab311df44dad4bbaba2456869b936cda [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 22:54:08.475198 1054 ts_tablet_manager.cc:1905] T 49c49116363b4a61846bdd1c724737da P ab311df44dad4bbaba2456869b936cda: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 22:54:08.475808 1052 tablet_replica.cc:330] T 49c49116363b4a61846bdd1c724737da P 071ed94da7b14cb1a61550a7af68ec6f: stopping tablet replica
I20260929 22:54:08.475922 1052 raft_consensus.cc:2229] T 49c49116363b4a61846bdd1c724737da P 071ed94da7b14cb1a61550a7af68ec6f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 22:54:08.476024 1052 raft_consensus.cc:2258] T 49c49116363b4a61846bdd1c724737da P 071ed94da7b14cb1a61550a7af68ec6f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 22:54:08.476413 1052 ts_tablet_manager.cc:1905] T 49c49116363b4a61846bdd1c724737da P 071ed94da7b14cb1a61550a7af68ec6f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 22:54:08.477190 1054 ts_tablet_manager.cc:1918] T 49c49116363b4a61846bdd1c724737da P ab311df44dad4bbaba2456869b936cda: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 22:54:08.477257 1054 log.cc:1196] T 49c49116363b4a61846bdd1c724737da P ab311df44dad4bbaba2456869b936cda: Deleting WAL directory at /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790722213368738-14927-0/minicluster-data/ts-2/wal/wals/49c49116363b4a61846bdd1c724737da
I20260929 22:54:08.477501 1054 ts_tablet_manager.cc:1939] T 49c49116363b4a61846bdd1c724737da P ab311df44dad4bbaba2456869b936cda: Deleting consensus metadata
I20260929 22:54:08.477885 540 catalog_manager.cc:4841] TS ab311df44dad4bbaba2456869b936cda (127.14.147.195:33065): tablet 49c49116363b4a61846bdd1c724737da (table test-table [id=15d87b0dbb1d442e9d2636c414a0b3fb]) successfully deleted
I20260929 22:54:08.477998 1052 ts_tablet_manager.cc:1918] T 49c49116363b4a61846bdd1c724737da P 071ed94da7b14cb1a61550a7af68ec6f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 22:54:08.478065 1052 log.cc:1196] T 49c49116363b4a61846bdd1c724737da P 071ed94da7b14cb1a61550a7af68ec6f: Deleting WAL directory at /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790722213368738-14927-0/minicluster-data/ts-1/wal/wals/49c49116363b4a61846bdd1c724737da
I20260929 22:54:08.478338 1052 ts_tablet_manager.cc:1939] T 49c49116363b4a61846bdd1c724737da P 071ed94da7b14cb1a61550a7af68ec6f: Deleting consensus metadata
I20260929 22:54:08.478724 542 catalog_manager.cc:4841] TS 071ed94da7b14cb1a61550a7af68ec6f (127.14.147.194:46351): tablet 49c49116363b4a61846bdd1c724737da (table test-table [id=15d87b0dbb1d442e9d2636c414a0b3fb]) successfully deleted
W20260929 22:54:08.479682 541 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.14.147.193:42539 (error 108)
I20260929 22:54:08.480134 1026 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:54:08.480007 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.147.193:42539 (local address 127.0.0.1:41836)
0929 22:54:08.480064 (+ 57us) negotiation.cc:107] Waiting for socket to connect
0929 22:54:08.480105 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.147.193:42539: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
I20260929 22:54:08.480123 14927 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhWOCLk/build/release/bin/kudu with pid 753
W20260929 22:54:08.481575 541 catalog_manager.cc:4584] TS 5ca5d8fcb5ea447ebc9554162325cc77 (127.14.147.193:42539): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 49c49116363b4a61846bdd1c724737da: Network error: Client connection negotiation failed: client connection to 127.14.147.193:42539: connect: Connection refused (error 111)
I20260929 22:54:08.484503 14927 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhWOCLk/build/release/bin/kudu with pid 887
I20260929 22:54:08.489526 14927 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhWOCLk/build/release/bin/kudu with pid 524
2026-09-29T22:54:08Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/2 (1802 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/3
2026-09-29T22:54:08Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T22:54:08Z Disabled control of system clock
I20260929 22:54:08.516093 14927 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhWOCLk/build/release/bin/kudu
/tmp/dist-test-taskhWOCLk/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790722213368738-14927-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790722213368738-14927-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790722213368738-14927-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790722213368738-14927-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.147.254:35717
--webserver_interface=127.14.147.254
--webserver_port=0
--builtin_ntp_servers=127.14.147.212:41905
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.147.254:35717
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260929 22:54:08.598696 1059 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 22:54:08.598953 1059 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 22:54:08.598975 1059 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 22:54:08.600209 1059 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 22:54:08.600255 1059 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 22:54:08.600265 1059 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 22:54:08.600277 1059 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 22:54:08.600288 1059 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 22:54:08.601675 1059 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.147.212:41905
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790722213368738-14927-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790722213368738-14927-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.14.147.254:35717
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.147.254:35717
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790722213368738-14927-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.14.147.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1059
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790722213368738-14927-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 21:58:54 UTC on ccycloud.quasar-xguoti.root.comops.site
build id 4681390
I20260929 22:54:08.601914 1059 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 22:54:08.602183 1059 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 22:54:08.606930 1065 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 22:54:08.607028 1067 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 22:54:08.607192 1064 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 22:54:08.607329 1059 server_base.cc:1034] running on GCE node
I20260929 22:54:08.607757 1059 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 22:54:08.608114 1059 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 22:54:08.609275 1059 hybrid_clock.cc:648] HybridClock initialized: now 1790722448609261 us; error 33 us; skew 500 ppm
I20260929 22:54:08.610589 1059 webserver.cc:460] Webserver started at http://127.14.147.254:33639/ using document root <none> and password file <none>
I20260929 22:54:08.610834 1059 fs_manager.cc:359] Metadata directory not provided
I20260929 22:54:08.610888 1059 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 22:54:08.611030 1059 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 22:54:08.612200 1059 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790722213368738-14927-0/minicluster-data/master-0/data/instance:
uuid: "4a2790239aea42bc845c0eb52f6c83a7"
format_stamp: "Formatted at 2026-09-29 22:54:08 on dist-test-slave-98mk"
server_key: "4af71ca760a8d8cf920f300315dbdbf2"
server_key_iv: "361807a5c7afb48ed3d52904f627d0a1"
server_key_version: "encryptionkey@0"
I20260929 22:54:08.612550 1059 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790722213368738-14927-0/minicluster-data/master-0/wal/instance:
uuid: "4a2790239aea42bc845c0eb52f6c83a7"
format_stamp: "Formatted at 2026-09-29 22:54:08 on dist-test-slave-98mk"
server_key: "4af71ca760a8d8cf920f300315dbdbf2"
server_key_iv: "361807a5c7afb48ed3d52904f627d0a1"
server_key_version: "encryptionkey@0"
I20260929 22:54:08.613850 1059 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260929 22:54:08.614566 1073 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 22:54:08.614748 1059 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260929 22:54:08.614831 1059 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790722213368738-14927-0/minicluster-data/master-0/data,/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790722213368738-14927-0/minicluster-data/master-0/wal
uuid: "4a2790239aea42bc845c0eb52f6c83a7"
format_stamp: "Formatted at 2026-09-29 22:54:08 on dist-test-slave-98mk"
server_key: "4af71ca760a8d8cf920f300315dbdbf2"
server_key_iv: "361807a5c7afb48ed3d52904f627d0a1"
server_key_version: "encryptionkey@0"
I20260929 22:54:08.614907 1059 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790722213368738-14927-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790722213368738-14927-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790722213368738-14927-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 22:54:08.624253 1059 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 22:54:08.624576 1059 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 22:54:08.624701 1059 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 22:54:08.628479 1059 rpc_server.cc:307] RPC server started. Bound to: 127.14.147.254:35717
I20260929 22:54:08.628509 1125 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.147.254:35717 every 8 connection(s)
I20260929 22:54:08.628857 1059 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790722213368738-14927-0/minicluster-data/master-0/data/info.pb
I20260929 22:54:08.629820 1126 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 22:54:08.630527 14927 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhWOCLk/build/release/bin/kudu as pid 1059
I20260929 22:54:08.630633 14927 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790722213368738-14927-0/minicluster-data/master-0/wal/instance
I20260929 22:54:08.630812 14927 external_mini_cluster.cc:1428] Setting key 60dd368d4a82f2e5b8251a293ff1f1d8
I20260929 22:54:08.632817 1126 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4a2790239aea42bc845c0eb52f6c83a7: Bootstrap starting.
I20260929 22:54:08.633406 1126 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4a2790239aea42bc845c0eb52f6c83a7: Neither blocks nor log segments found. Creating new log.
I20260929 22:54:08.633702 1126 log.cc:824] T 00000000000000000000000000000000 P 4a2790239aea42bc845c0eb52f6c83a7: Log is configured to *not* fsync() on all Append() calls
I20260929 22:54:08.634421 1126 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4a2790239aea42bc845c0eb52f6c83a7: No bootstrap required, opened a new log
I20260929 22:54:08.636327 1129 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:54:08.631344 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54956 (local address 127.14.147.254:35717)
0929 22:54:08.631564 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0929 22:54:08.631572 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 22:54:08.631597 (+ 25us) server_negotiation.cc:407] Connection header received
0929 22:54:08.632043 (+ 446us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 22:54:08.632050 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 22:54:08.632112 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 22:54:08.632227 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0929 22:54:08.632456 (+ 229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:54:08.633331 (+ 875us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:54:08.634262 (+ 931us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:54:08.634686 (+ 424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:54:08.634782 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 22:54:08.635538 (+ 756us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 22:54:08.635542 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 22:54:08.635542 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 22:54:08.635705 (+ 163us) server_negotiation.cc:922] calling sasl_server_start()
0929 22:54:08.635738 (+ 33us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 22:54:08.635748 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 22:54:08.635806 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0929 22:54:08.635957 (+ 151us) server_negotiation.cc:299] Negotiation successful
0929 22:54:08.636047 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":155,"thread_start_us":96,"threads_started":1}
I20260929 22:54:08.636762 1126 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4a2790239aea42bc845c0eb52f6c83a7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4a2790239aea42bc845c0eb52f6c83a7" member_type: VOTER last_known_addr { host: "127.14.147.254" port: 35717 } }
I20260929 22:54:08.636925 1126 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4a2790239aea42bc845c0eb52f6c83a7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 22:54:08.636948 1126 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4a2790239aea42bc845c0eb52f6c83a7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4a2790239aea42bc845c0eb52f6c83a7, State: Initialized, Role: FOLLOWER
I20260929 22:54:08.637080 1126 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4a2790239aea42bc845c0eb52f6c83a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4a2790239aea42bc845c0eb52f6c83a7" member_type: VOTER last_known_addr { host: "127.14.147.254" port: 35717 } }
I20260929 22:54:08.637154 1126 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4a2790239aea42bc845c0eb52f6c83a7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 22:54:08.637182 1126 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4a2790239aea42bc845c0eb52f6c83a7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 22:54:08.637230 1126 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4a2790239aea42bc845c0eb52f6c83a7 [term 0 FOLLOWER]: Advancing to term 1
I20260929 22:54:08.637825 1126 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4a2790239aea42bc845c0eb52f6c83a7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4a2790239aea42bc845c0eb52f6c83a7" member_type: VOTER last_known_addr { host: "127.14.147.254" port: 35717 } }
I20260929 22:54:08.637916 1126 leader_election.cc:302] T 00000000000000000000000000000000 P 4a2790239aea42bc845c0eb52f6c83a7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4a2790239aea42bc845c0eb52f6c83a7; no voters:
I20260929 22:54:08.638047 1126 leader_election.cc:288] T 00000000000000000000000000000000 P 4a2790239aea42bc845c0eb52f6c83a7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 22:54:08.638130 1131 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4a2790239aea42bc845c0eb52f6c83a7 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 22:54:08.638284 1131 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4a2790239aea42bc845c0eb52f6c83a7 [term 1 LEADER]: Becoming Leader. State: Replica: 4a2790239aea42bc845c0eb52f6c83a7, State: Running, Role: LEADER
I20260929 22:54:08.638315 1126 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4a2790239aea42bc845c0eb52f6c83a7 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 22:54:08.638374 1131 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4a2790239aea42bc845c0eb52f6c83a7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4a2790239aea42bc845c0eb52f6c83a7" member_type: VOTER last_known_addr { host: "127.14.147.254" port: 35717 } }
I20260929 22:54:08.638762 1133 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4a2790239aea42bc845c0eb52f6c83a7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4a2790239aea42bc845c0eb52f6c83a7. Latest consensus state: current_term: 1 leader_uuid: "4a2790239aea42bc845c0eb52f6c83a7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4a2790239aea42bc845c0eb52f6c83a7" member_type: VOTER last_known_addr { host: "127.14.147.254" port: 35717 } } }
I20260929 22:54:08.638849 1133 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4a2790239aea42bc845c0eb52f6c83a7 [sys.catalog]: This master's current role is: LEADER
I20260929 22:54:08.638993 1132 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4a2790239aea42bc845c0eb52f6c83a7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4a2790239aea42bc845c0eb52f6c83a7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4a2790239aea42bc845c0eb52f6c83a7" member_type: VOTER last_known_addr { host: "127.14.147.254" port: 35717 } } }
I20260929 22:54:08.639038 1140 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 22:54:08.639055 1132 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4a2790239aea42bc845c0eb52f6c83a7 [sys.catalog]: This master's current role is: LEADER
I20260929 22:54:08.639599 1140 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 22:54:08.640990 1140 catalog_manager.cc:1347] Generated new cluster ID: 1f713a4bbc074a64846af7518710a765
I20260929 22:54:08.641049 1140 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 22:54:08.654878 1140 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 22:54:08.655797 1140 catalog_manager.cc:1504] Loading token signing keys...
I20260929 22:54:08.663156 1140 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4a2790239aea42bc845c0eb52f6c83a7: Generated new TSK 0
I20260929 22:54:08.663373 1140 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 22:54:08.673040 14927 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhWOCLk/build/release/bin/kudu
/tmp/dist-test-taskhWOCLk/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790722213368738-14927-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790722213368738-14927-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790722213368738-14927-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790722213368738-14927-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.147.193:0
--local_ip_for_outbound_sockets=127.14.147.193
--webserver_interface=127.14.147.193
--webserver_port=0
--tserver_master_addrs=127.14.147.254:35717
--builtin_ntp_servers=127.14.147.212:41905
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260929 22:54:08.760483 1150 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 22:54:08.760702 1150 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 22:54:08.760728 1150 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 22:54:08.761999 1150 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 22:54:08.762044 1150 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 22:54:08.762071 1150 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.147.193
I20260929 22:54:08.763510 1150 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.147.212:41905
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790722213368738-14927-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790722213368738-14927-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.147.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790722213368738-14927-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.14.147.193
--webserver_port=0
--tserver_master_addrs=127.14.147.254:35717
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1150
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.147.193
--log_dir=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790722213368738-14927-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 21:58:54 UTC on ccycloud.quasar-xguoti.root.comops.site
build id 4681390
I20260929 22:54:08.763762 1150 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 22:54:08.764058 1150 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 22:54:08.769518 1156 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 22:54:08.769538 1158 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 22:54:08.769515 1155 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 22:54:08.769731 1150 server_base.cc:1034] running on GCE node
I20260929 22:54:08.769948 1150 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 22:54:08.770196 1150 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 22:54:08.771382 1150 hybrid_clock.cc:648] HybridClock initialized: now 1790722448771363 us; error 32 us; skew 500 ppm
I20260929 22:54:08.772766 1150 webserver.cc:460] Webserver started at http://127.14.147.193:34067/ using document root <none> and password file <none>
I20260929 22:54:08.772993 1150 fs_manager.cc:359] Metadata directory not provided
I20260929 22:54:08.773042 1150 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 22:54:08.773172 1150 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 22:54:08.774286 1150 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790722213368738-14927-0/minicluster-data/ts-0/data/instance:
uuid: "8d3e3ac5cd67441aa4ae1aa81d910722"
format_stamp: "Formatted at 2026-09-29 22:54:08 on dist-test-slave-98mk"
server_key: "bd82cbd0cce9f077dd46e24635789f00"
server_key_iv: "072f94d632852765c07c3bdeecde61a5"
server_key_version: "encryptionkey@0"
I20260929 22:54:08.774621 1150 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790722213368738-14927-0/minicluster-data/ts-0/wal/instance:
uuid: "8d3e3ac5cd67441aa4ae1aa81d910722"
format_stamp: "Formatted at 2026-09-29 22:54:08 on dist-test-slave-98mk"
server_key: "bd82cbd0cce9f077dd46e24635789f00"
server_key_iv: "072f94d632852765c07c3bdeecde61a5"
server_key_version: "encryptionkey@0"
I20260929 22:54:08.776037 1150 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260929 22:54:08.776767 1164 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 22:54:08.776938 1150 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260929 22:54:08.777020 1150 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790722213368738-14927-0/minicluster-data/ts-0/data,/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790722213368738-14927-0/minicluster-data/ts-0/wal
uuid: "8d3e3ac5cd67441aa4ae1aa81d910722"
format_stamp: "Formatted at 2026-09-29 22:54:08 on dist-test-slave-98mk"
server_key: "bd82cbd0cce9f077dd46e24635789f00"
server_key_iv: "072f94d632852765c07c3bdeecde61a5"
server_key_version: "encryptionkey@0"
I20260929 22:54:08.777082 1150 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790722213368738-14927-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790722213368738-14927-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790722213368738-14927-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 22:54:08.789439 1150 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 22:54:08.789762 1150 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 22:54:08.789901 1150 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 22:54:08.790128 1150 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 22:54:08.790460 1150 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 22:54:08.790496 1150 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 22:54:08.790526 1150 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 22:54:08.790542 1150 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 22:54:08.796221 1150 rpc_server.cc:307] RPC server started. Bound to: 127.14.147.193:40941
I20260929 22:54:08.796273 1277 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.147.193:40941 every 8 connection(s)
I20260929 22:54:08.796615 1150 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790722213368738-14927-0/minicluster-data/ts-0/data/info.pb
I20260929 22:54:08.799296 14927 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhWOCLk/build/release/bin/kudu as pid 1150
I20260929 22:54:08.799398 14927 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790722213368738-14927-0/minicluster-data/ts-0/wal/instance
I20260929 22:54:08.799631 14927 external_mini_cluster.cc:1428] Setting key 97a8e1fae6c3da5df76cc86c1f52b52a
I20260929 22:54:08.800652 14927 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhWOCLk/build/release/bin/kudu
/tmp/dist-test-taskhWOCLk/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790722213368738-14927-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790722213368738-14927-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790722213368738-14927-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790722213368738-14927-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.147.194:0
--local_ip_for_outbound_sockets=127.14.147.194
--webserver_interface=127.14.147.194
--webserver_port=0
--tserver_master_addrs=127.14.147.254:35717
--builtin_ntp_servers=127.14.147.212:41905
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260929 22:54:08.802389 1129 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:54:08.797346 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.147.193:57741 (local address 127.14.147.254:35717)
0929 22:54:08.797405 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0929 22:54:08.797407 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 22:54:08.797604 (+ 197us) server_negotiation.cc:407] Connection header received
0929 22:54:08.798134 (+ 530us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 22:54:08.798136 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 22:54:08.798152 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 22:54:08.798206 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0929 22:54:08.798879 (+ 673us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:54:08.799393 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:54:08.800807 (+ 1414us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:54:08.801057 (+ 250us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:54:08.801100 (+ 43us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 22:54:08.802105 (+ 1005us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 22:54:08.802107 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 22:54:08.802107 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 22:54:08.802114 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0929 22:54:08.802133 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 22:54:08.802137 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 22:54:08.802194 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0929 22:54:08.802267 (+ 73us) server_negotiation.cc:299] Negotiation successful
0929 22:54:08.802305 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260929 22:54:08.802556 1280 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:54:08.797319 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.147.254:35717 (local address 127.14.147.193:57741)
0929 22:54:08.797485 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0929 22:54:08.797519 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0929 22:54:08.798031 (+ 512us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 22:54:08.798277 (+ 246us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 22:54:08.798281 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 22:54:08.798289 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0929 22:54:08.798787 (+ 498us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 22:54:08.798791 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 22:54:08.799529 (+ 738us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 22:54:08.799530 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 22:54:08.800684 (+ 1154us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 22:54:08.800688 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 22:54:08.801195 (+ 507us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 22:54:08.801196 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 22:54:08.801366 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 22:54:08.801921 (+ 555us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 22:54:08.801921 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0929 22:54:08.801966 (+ 45us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 22:54:08.801999 (+ 33us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 22:54:08.802200 (+ 201us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 22:54:08.802204 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 22:54:08.802207 (+ 3us) client_negotiation.cc:769] Sending connection context
0929 22:54:08.802258 (+ 51us) client_negotiation.cc:240] Negotiation successful
0929 22:54:08.802355 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":106,"thread_start_us":70,"threads_started":1}
I20260929 22:54:08.803063 1278 heartbeater.cc:347] Connected to a master server at 127.14.147.254:35717
I20260929 22:54:08.803161 1278 heartbeater.cc:464] Registering TS with master...
I20260929 22:54:08.803352 1278 heartbeater.cc:511] Master 127.14.147.254:35717 requested a full tablet report, sending...
I20260929 22:54:08.803895 1090 ts_manager.cc:194] Registered new tserver with Master: 8d3e3ac5cd67441aa4ae1aa81d910722 (127.14.147.193:40941)
I20260929 22:54:08.804569 1090 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.14.147.193:57741
W20260929 22:54:08.882140 1281 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 22:54:08.882366 1281 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 22:54:08.882388 1281 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 22:54:08.883654 1281 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 22:54:08.883701 1281 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 22:54:08.883729 1281 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.147.194
I20260929 22:54:08.885145 1281 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.147.212:41905
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790722213368738-14927-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790722213368738-14927-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.147.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790722213368738-14927-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.14.147.194
--webserver_port=0
--tserver_master_addrs=127.14.147.254:35717
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1281
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.147.194
--log_dir=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790722213368738-14927-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 21:58:54 UTC on ccycloud.quasar-xguoti.root.comops.site
build id 4681390
I20260929 22:54:08.885386 1281 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 22:54:08.885785 1281 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 22:54:08.891347 1287 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 22:54:08.891335 1289 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 22:54:08.891332 1281 server_base.cc:1034] running on GCE node
W20260929 22:54:08.891332 1286 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 22:54:08.891755 1281 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 22:54:08.891979 1281 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 22:54:08.893118 1281 hybrid_clock.cc:648] HybridClock initialized: now 1790722448893102 us; error 32 us; skew 500 ppm
I20260929 22:54:08.894364 1281 webserver.cc:460] Webserver started at http://127.14.147.194:42745/ using document root <none> and password file <none>
I20260929 22:54:08.894580 1281 fs_manager.cc:359] Metadata directory not provided
I20260929 22:54:08.894629 1281 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 22:54:08.894762 1281 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 22:54:08.895876 1281 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790722213368738-14927-0/minicluster-data/ts-1/data/instance:
uuid: "7ef432ec71794579a385c250eaa5b68d"
format_stamp: "Formatted at 2026-09-29 22:54:08 on dist-test-slave-98mk"
server_key: "80fb3d5b6f0bc67ebcc0c005f47c28fb"
server_key_iv: "bc88918ca4450d0c73cb89eda61dca1a"
server_key_version: "encryptionkey@0"
I20260929 22:54:08.896198 1281 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790722213368738-14927-0/minicluster-data/ts-1/wal/instance:
uuid: "7ef432ec71794579a385c250eaa5b68d"
format_stamp: "Formatted at 2026-09-29 22:54:08 on dist-test-slave-98mk"
server_key: "80fb3d5b6f0bc67ebcc0c005f47c28fb"
server_key_iv: "bc88918ca4450d0c73cb89eda61dca1a"
server_key_version: "encryptionkey@0"
I20260929 22:54:08.897444 1281 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260929 22:54:08.898212 1295 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 22:54:08.898389 1281 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260929 22:54:08.898468 1281 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790722213368738-14927-0/minicluster-data/ts-1/data,/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790722213368738-14927-0/minicluster-data/ts-1/wal
uuid: "7ef432ec71794579a385c250eaa5b68d"
format_stamp: "Formatted at 2026-09-29 22:54:08 on dist-test-slave-98mk"
server_key: "80fb3d5b6f0bc67ebcc0c005f47c28fb"
server_key_iv: "bc88918ca4450d0c73cb89eda61dca1a"
server_key_version: "encryptionkey@0"
I20260929 22:54:08.898533 1281 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790722213368738-14927-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790722213368738-14927-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790722213368738-14927-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 22:54:08.908238 1281 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 22:54:08.908529 1281 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 22:54:08.908653 1281 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 22:54:08.908885 1281 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 22:54:08.909211 1281 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 22:54:08.909245 1281 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 22:54:08.909273 1281 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 22:54:08.909291 1281 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 22:54:08.914860 1281 rpc_server.cc:307] RPC server started. Bound to: 127.14.147.194:44675
I20260929 22:54:08.914899 1408 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.147.194:44675 every 8 connection(s)
I20260929 22:54:08.915222 1281 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790722213368738-14927-0/minicluster-data/ts-1/data/info.pb
I20260929 22:54:08.916272 14927 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhWOCLk/build/release/bin/kudu as pid 1281
I20260929 22:54:08.916365 14927 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790722213368738-14927-0/minicluster-data/ts-1/wal/instance
I20260929 22:54:08.916567 14927 external_mini_cluster.cc:1428] Setting key aad117714521ec5496eaea2fde5602d1
I20260929 22:54:08.917611 14927 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhWOCLk/build/release/bin/kudu
/tmp/dist-test-taskhWOCLk/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790722213368738-14927-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790722213368738-14927-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790722213368738-14927-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790722213368738-14927-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.147.195:0
--local_ip_for_outbound_sockets=127.14.147.195
--webserver_interface=127.14.147.195
--webserver_port=0
--tserver_master_addrs=127.14.147.254:35717
--builtin_ntp_servers=127.14.147.212:41905
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260929 22:54:08.920928 1129 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:54:08.915961 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.147.194:54625 (local address 127.14.147.254:35717)
0929 22:54:08.916021 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0929 22:54:08.916023 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 22:54:08.916270 (+ 247us) server_negotiation.cc:407] Connection header received
0929 22:54:08.916903 (+ 633us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 22:54:08.916904 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 22:54:08.916916 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 22:54:08.916970 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0929 22:54:08.917866 (+ 896us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:54:08.918457 (+ 591us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:54:08.919564 (+ 1107us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:54:08.919758 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:54:08.919818 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 22:54:08.920681 (+ 863us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 22:54:08.920683 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 22:54:08.920683 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 22:54:08.920687 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0929 22:54:08.920703 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 22:54:08.920707 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 22:54:08.920750 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0929 22:54:08.920840 (+ 90us) server_negotiation.cc:299] Negotiation successful
0929 22:54:08.920866 (+ 26us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260929 22:54:08.921175 1411 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:54:08.915935 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.147.254:35717 (local address 127.14.147.194:54625)
0929 22:54:08.916120 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0929 22:54:08.916168 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0929 22:54:08.916823 (+ 655us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 22:54:08.917049 (+ 226us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 22:54:08.917053 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 22:54:08.917073 (+ 20us) client_negotiation.cc:189] Negotiated authn=SASL
0929 22:54:08.917779 (+ 706us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 22:54:08.917786 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 22:54:08.918546 (+ 760us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 22:54:08.918547 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 22:54:08.919481 (+ 934us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 22:54:08.919484 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 22:54:08.919832 (+ 348us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 22:54:08.919832 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 22:54:08.919980 (+ 148us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 22:54:08.920521 (+ 541us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 22:54:08.920522 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0929 22:54:08.920560 (+ 38us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 22:54:08.920589 (+ 29us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 22:54:08.920768 (+ 179us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 22:54:08.920772 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 22:54:08.920775 (+ 3us) client_negotiation.cc:769] Sending connection context
0929 22:54:08.920842 (+ 67us) client_negotiation.cc:240] Negotiation successful
0929 22:54:08.920951 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":75,"threads_started":1}
I20260929 22:54:08.921629 1409 heartbeater.cc:347] Connected to a master server at 127.14.147.254:35717
I20260929 22:54:08.921728 1409 heartbeater.cc:464] Registering TS with master...
I20260929 22:54:08.921908 1409 heartbeater.cc:511] Master 127.14.147.254:35717 requested a full tablet report, sending...
I20260929 22:54:08.922282 1090 ts_manager.cc:194] Registered new tserver with Master: 7ef432ec71794579a385c250eaa5b68d (127.14.147.194:44675)
I20260929 22:54:08.922631 1090 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.14.147.194:54625
W20260929 22:54:08.997272 1412 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 22:54:08.997483 1412 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 22:54:08.997504 1412 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 22:54:08.998661 1412 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 22:54:08.998703 1412 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 22:54:08.998728 1412 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.147.195
I20260929 22:54:09.000109 1412 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.147.212:41905
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790722213368738-14927-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790722213368738-14927-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.147.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790722213368738-14927-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.14.147.195
--webserver_port=0
--tserver_master_addrs=127.14.147.254:35717
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1412
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.147.195
--log_dir=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790722213368738-14927-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 21:58:54 UTC on ccycloud.quasar-xguoti.root.comops.site
build id 4681390
I20260929 22:54:09.000351 1412 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 22:54:09.000634 1412 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 22:54:09.005489 1420 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 22:54:09.005525 1418 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 22:54:09.005599 1412 server_base.cc:1034] running on GCE node
W20260929 22:54:09.005496 1417 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 22:54:09.005903 1412 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 22:54:09.006206 1412 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 22:54:09.007387 1412 hybrid_clock.cc:648] HybridClock initialized: now 1790722449007373 us; error 24 us; skew 500 ppm
I20260929 22:54:09.008730 1412 webserver.cc:460] Webserver started at http://127.14.147.195:43439/ using document root <none> and password file <none>
I20260929 22:54:09.008975 1412 fs_manager.cc:359] Metadata directory not provided
I20260929 22:54:09.009032 1412 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 22:54:09.009147 1412 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 22:54:09.010216 1412 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790722213368738-14927-0/minicluster-data/ts-2/data/instance:
uuid: "450311e81b584aec8d79736ebd48a766"
format_stamp: "Formatted at 2026-09-29 22:54:09 on dist-test-slave-98mk"
server_key: "545e76bdd32e34d968208a6d02e7673a"
server_key_iv: "f39a9c2f8a05db05437dc31ce76dd6d7"
server_key_version: "encryptionkey@0"
I20260929 22:54:09.010546 1412 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790722213368738-14927-0/minicluster-data/ts-2/wal/instance:
uuid: "450311e81b584aec8d79736ebd48a766"
format_stamp: "Formatted at 2026-09-29 22:54:09 on dist-test-slave-98mk"
server_key: "545e76bdd32e34d968208a6d02e7673a"
server_key_iv: "f39a9c2f8a05db05437dc31ce76dd6d7"
server_key_version: "encryptionkey@0"
I20260929 22:54:09.011874 1412 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260929 22:54:09.012573 1426 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 22:54:09.012758 1412 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260929 22:54:09.012846 1412 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790722213368738-14927-0/minicluster-data/ts-2/data,/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790722213368738-14927-0/minicluster-data/ts-2/wal
uuid: "450311e81b584aec8d79736ebd48a766"
format_stamp: "Formatted at 2026-09-29 22:54:09 on dist-test-slave-98mk"
server_key: "545e76bdd32e34d968208a6d02e7673a"
server_key_iv: "f39a9c2f8a05db05437dc31ce76dd6d7"
server_key_version: "encryptionkey@0"
I20260929 22:54:09.012923 1412 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790722213368738-14927-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790722213368738-14927-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790722213368738-14927-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 22:54:09.021428 1412 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 22:54:09.021734 1412 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 22:54:09.021855 1412 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 22:54:09.022076 1412 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 22:54:09.022384 1412 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 22:54:09.022418 1412 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 22:54:09.022447 1412 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 22:54:09.022465 1412 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 22:54:09.027964 1412 rpc_server.cc:307] RPC server started. Bound to: 127.14.147.195:39383
I20260929 22:54:09.028049 1539 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.147.195:39383 every 8 connection(s)
I20260929 22:54:09.028348 1412 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790722213368738-14927-0/minicluster-data/ts-2/data/info.pb
I20260929 22:54:09.033070 14927 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhWOCLk/build/release/bin/kudu as pid 1412
I20260929 22:54:09.033164 14927 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790722213368738-14927-0/minicluster-data/ts-2/wal/instance
I20260929 22:54:09.033329 14927 external_mini_cluster.cc:1428] Setting key 7e745c97f9041ef3420aa04728cd4d10
I20260929 22:54:09.037283 1129 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:54:09.029371 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.147.195:59753 (local address 127.14.147.254:35717)
0929 22:54:09.029477 (+ 106us) server_negotiation.cc:206] Beginning negotiation
0929 22:54:09.029480 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 22:54:09.029719 (+ 239us) server_negotiation.cc:407] Connection header received
0929 22:54:09.030363 (+ 644us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 22:54:09.030364 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 22:54:09.030379 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 22:54:09.030443 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0929 22:54:09.031380 (+ 937us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:54:09.031927 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:54:09.032973 (+ 1046us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:54:09.033204 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:54:09.033283 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 22:54:09.036999 (+ 3716us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 22:54:09.037001 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 22:54:09.037001 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 22:54:09.037007 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0929 22:54:09.037026 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 22:54:09.037032 (+ 6us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 22:54:09.037075 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0929 22:54:09.037158 (+ 83us) server_negotiation.cc:299] Negotiation successful
0929 22:54:09.037195 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260929 22:54:09.037534 1542 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:54:09.029268 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.147.254:35717 (local address 127.14.147.195:59753)
0929 22:54:09.029558 (+ 290us) negotiation.cc:107] Waiting for socket to connect
0929 22:54:09.029606 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0929 22:54:09.030247 (+ 641us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 22:54:09.030626 (+ 379us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 22:54:09.030632 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 22:54:09.030644 (+ 12us) client_negotiation.cc:189] Negotiated authn=SASL
0929 22:54:09.031197 (+ 553us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 22:54:09.031203 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 22:54:09.032026 (+ 823us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 22:54:09.032027 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 22:54:09.032856 (+ 829us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 22:54:09.032860 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 22:54:09.033305 (+ 445us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 22:54:09.033306 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 22:54:09.033481 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 22:54:09.036802 (+ 3321us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 22:54:09.036803 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0929 22:54:09.036856 (+ 53us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 22:54:09.036898 (+ 42us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 22:54:09.037085 (+ 187us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 22:54:09.037092 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 22:54:09.037095 (+ 3us) client_negotiation.cc:769] Sending connection context
0929 22:54:09.037163 (+ 68us) client_negotiation.cc:240] Negotiation successful
0929 22:54:09.037290 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":171,"thread_start_us":99,"threads_started":1}
I20260929 22:54:09.038007 1540 heartbeater.cc:347] Connected to a master server at 127.14.147.254:35717
I20260929 22:54:09.038112 1540 heartbeater.cc:464] Registering TS with master...
I20260929 22:54:09.038328 1540 heartbeater.cc:511] Master 127.14.147.254:35717 requested a full tablet report, sending...
I20260929 22:54:09.038722 1090 ts_manager.cc:194] Registered new tserver with Master: 450311e81b584aec8d79736ebd48a766 (127.14.147.195:39383)
I20260929 22:54:09.039096 1090 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.14.147.195:59753
I20260929 22:54:09.045214 14927 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 22:54:09.049423 1129 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:54:09.046546 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54958 (local address 127.14.147.254:35717)
0929 22:54:09.046605 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0929 22:54:09.046607 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 22:54:09.046690 (+ 83us) server_negotiation.cc:407] Connection header received
0929 22:54:09.046744 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 22:54:09.046745 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 22:54:09.046753 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 22:54:09.046811 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0929 22:54:09.046951 (+ 140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:54:09.047460 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:54:09.048214 (+ 754us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:54:09.048405 (+ 191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:54:09.048458 (+ 53us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 22:54:09.049073 (+ 615us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 22:54:09.049074 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 22:54:09.049074 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 22:54:09.049078 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0929 22:54:09.049092 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 22:54:09.049096 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 22:54:09.049161 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0929 22:54:09.049347 (+ 186us) server_negotiation.cc:299] Negotiation successful
0929 22:54:09.049359 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260929 22:54:09.051155 1090 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54958:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 22:54:09.052850 1090 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 22:54:09.058848 1553 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:54:09.055245 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36792 (local address 127.14.147.195:39383)
0929 22:54:09.055409 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0929 22:54:09.055435 (+ 26us) server_negotiation.cc:399] Waiting for connection header
0929 22:54:09.055445 (+ 10us) server_negotiation.cc:407] Connection header received
0929 22:54:09.055481 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 22:54:09.055483 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 22:54:09.055497 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 22:54:09.055567 (+ 70us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 22:54:09.055701 (+ 134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:54:09.056516 (+ 815us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:54:09.057903 (+ 1387us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:54:09.058439 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:54:09.058525 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 22:54:09.058621 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0929 22:54:09.058706 (+ 85us) server_negotiation.cc:299] Negotiation successful
0929 22:54:09.058750 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":131,"thread_start_us":87,"threads_started":1}
I20260929 22:54:09.058835 1548 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:54:09.054801 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.147.195:39383 (local address 127.0.0.1:36792)
0929 22:54:09.055067 (+ 266us) negotiation.cc:107] Waiting for socket to connect
0929 22:54:09.055077 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0929 22:54:09.055129 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 22:54:09.055575 (+ 446us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 22:54:09.055576 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 22:54:09.055578 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 22:54:09.055625 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 22:54:09.055627 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 22:54:09.056594 (+ 967us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 22:54:09.056595 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 22:54:09.057811 (+ 1216us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 22:54:09.057814 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 22:54:09.058535 (+ 721us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 22:54:09.058536 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 22:54:09.058616 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 22:54:09.058629 (+ 13us) client_negotiation.cc:769] Sending connection context
0929 22:54:09.058694 (+ 65us) client_negotiation.cc:240] Negotiation successful
0929 22:54:09.058750 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":251,"thread_start_us":71,"threads_started":1}
I20260929 22:54:09.058965 1549 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:54:09.054800 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.147.194:44675 (local address 127.0.0.1:50308)
0929 22:54:09.054965 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0929 22:54:09.055002 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0929 22:54:09.055072 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 22:54:09.055410 (+ 338us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 22:54:09.055434 (+ 24us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 22:54:09.055437 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 22:54:09.055516 (+ 79us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 22:54:09.055518 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 22:54:09.056912 (+ 1394us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 22:54:09.056913 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 22:54:09.057931 (+ 1018us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 22:54:09.057933 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 22:54:09.058779 (+ 846us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 22:54:09.058780 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 22:54:09.058833 (+ 53us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 22:54:09.058835 (+ 2us) client_negotiation.cc:769] Sending connection context
0929 22:54:09.058860 (+ 25us) client_negotiation.cc:240] Negotiation successful
0929 22:54:09.058892 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":108,"threads_started":1}
I20260929 22:54:09.058965 1550 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:54:09.054804 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.147.193:40941 (local address 127.0.0.1:51470)
0929 22:54:09.055177 (+ 373us) negotiation.cc:107] Waiting for socket to connect
0929 22:54:09.055186 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0929 22:54:09.055220 (+ 34us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 22:54:09.055657 (+ 437us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 22:54:09.055658 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 22:54:09.055659 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 22:54:09.055688 (+ 29us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 22:54:09.055689 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 22:54:09.057005 (+ 1316us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 22:54:09.057006 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 22:54:09.057780 (+ 774us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 22:54:09.057783 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 22:54:09.058690 (+ 907us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 22:54:09.058691 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 22:54:09.058750 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 22:54:09.058756 (+ 6us) client_negotiation.cc:769] Sending connection context
0929 22:54:09.058784 (+ 28us) client_negotiation.cc:240] Negotiation successful
0929 22:54:09.058825 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":359,"thread_start_us":92,"threads_started":1}
I20260929 22:54:09.059401 1552 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:54:09.055075 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50308 (local address 127.14.147.194:44675)
0929 22:54:09.055255 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0929 22:54:09.055258 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 22:54:09.055267 (+ 9us) server_negotiation.cc:407] Connection header received
0929 22:54:09.055304 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 22:54:09.055305 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 22:54:09.055317 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 22:54:09.055386 (+ 69us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 22:54:09.055598 (+ 212us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:54:09.056817 (+ 1219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:54:09.058007 (+ 1190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:54:09.058701 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:54:09.059126 (+ 425us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 22:54:09.059230 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0929 22:54:09.059257 (+ 27us) server_negotiation.cc:299] Negotiation successful
0929 22:54:09.059301 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":149,"thread_start_us":81,"threads_started":1}
I20260929 22:54:09.059453 1551 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:54:09.055068 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51470 (local address 127.14.147.193:40941)
0929 22:54:09.055524 (+ 456us) server_negotiation.cc:206] Beginning negotiation
0929 22:54:09.055527 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 22:54:09.055536 (+ 9us) server_negotiation.cc:407] Connection header received
0929 22:54:09.055571 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 22:54:09.055573 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 22:54:09.055586 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 22:54:09.055775 (+ 189us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 22:54:09.055847 (+ 72us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:54:09.056946 (+ 1099us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:54:09.057882 (+ 936us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:54:09.058591 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:54:09.059210 (+ 619us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 22:54:09.059298 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0929 22:54:09.059319 (+ 21us) server_negotiation.cc:299] Negotiation successful
0929 22:54:09.059357 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":422,"thread_start_us":79,"threads_started":1}
I20260929 22:54:09.059773 1474 tablet_service.cc:1467] Processing CreateTablet for tablet 1d1f4f04dca34d158952f455c4c947f4 (DEFAULT_TABLE table=test-table [id=5e9ea8225a7a4d2492246854be2a539e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 22:54:09.060110 1474 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1d1f4f04dca34d158952f455c4c947f4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 22:54:09.060029 1343 tablet_service.cc:1467] Processing CreateTablet for tablet 1d1f4f04dca34d158952f455c4c947f4 (DEFAULT_TABLE table=test-table [id=5e9ea8225a7a4d2492246854be2a539e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 22:54:09.060026 1212 tablet_service.cc:1467] Processing CreateTablet for tablet 1d1f4f04dca34d158952f455c4c947f4 (DEFAULT_TABLE table=test-table [id=5e9ea8225a7a4d2492246854be2a539e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 22:54:09.060343 1212 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1d1f4f04dca34d158952f455c4c947f4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 22:54:09.060343 1343 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1d1f4f04dca34d158952f455c4c947f4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 22:54:09.062906 1554 tablet_bootstrap.cc:492] T 1d1f4f04dca34d158952f455c4c947f4 P 450311e81b584aec8d79736ebd48a766: Bootstrap starting.
I20260929 22:54:09.063560 1554 tablet_bootstrap.cc:654] T 1d1f4f04dca34d158952f455c4c947f4 P 450311e81b584aec8d79736ebd48a766: Neither blocks nor log segments found. Creating new log.
I20260929 22:54:09.063596 1556 tablet_bootstrap.cc:492] T 1d1f4f04dca34d158952f455c4c947f4 P 7ef432ec71794579a385c250eaa5b68d: Bootstrap starting.
I20260929 22:54:09.064031 1554 log.cc:824] T 1d1f4f04dca34d158952f455c4c947f4 P 450311e81b584aec8d79736ebd48a766: Log is configured to *not* fsync() on all Append() calls
I20260929 22:54:09.064191 1556 tablet_bootstrap.cc:654] T 1d1f4f04dca34d158952f455c4c947f4 P 7ef432ec71794579a385c250eaa5b68d: Neither blocks nor log segments found. Creating new log.
I20260929 22:54:09.064460 1556 log.cc:824] T 1d1f4f04dca34d158952f455c4c947f4 P 7ef432ec71794579a385c250eaa5b68d: Log is configured to *not* fsync() on all Append() calls
I20260929 22:54:09.064771 1554 tablet_bootstrap.cc:492] T 1d1f4f04dca34d158952f455c4c947f4 P 450311e81b584aec8d79736ebd48a766: No bootstrap required, opened a new log
I20260929 22:54:09.064786 1555 tablet_bootstrap.cc:492] T 1d1f4f04dca34d158952f455c4c947f4 P 8d3e3ac5cd67441aa4ae1aa81d910722: Bootstrap starting.
I20260929 22:54:09.064838 1554 ts_tablet_manager.cc:1397] T 1d1f4f04dca34d158952f455c4c947f4 P 450311e81b584aec8d79736ebd48a766: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260929 22:54:09.065119 1556 tablet_bootstrap.cc:492] T 1d1f4f04dca34d158952f455c4c947f4 P 7ef432ec71794579a385c250eaa5b68d: No bootstrap required, opened a new log
I20260929 22:54:09.065181 1556 ts_tablet_manager.cc:1397] T 1d1f4f04dca34d158952f455c4c947f4 P 7ef432ec71794579a385c250eaa5b68d: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260929 22:54:09.065378 1555 tablet_bootstrap.cc:654] T 1d1f4f04dca34d158952f455c4c947f4 P 8d3e3ac5cd67441aa4ae1aa81d910722: Neither blocks nor log segments found. Creating new log.
I20260929 22:54:09.065697 1555 log.cc:824] T 1d1f4f04dca34d158952f455c4c947f4 P 8d3e3ac5cd67441aa4ae1aa81d910722: Log is configured to *not* fsync() on all Append() calls
I20260929 22:54:09.067198 1554 raft_consensus.cc:348] T 1d1f4f04dca34d158952f455c4c947f4 P 450311e81b584aec8d79736ebd48a766 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "450311e81b584aec8d79736ebd48a766" member_type: VOTER last_known_addr { host: "127.14.147.195" port: 39383 } } peers { permanent_uuid: "8d3e3ac5cd67441aa4ae1aa81d910722" member_type: VOTER last_known_addr { host: "127.14.147.193" port: 40941 } } peers { permanent_uuid: "7ef432ec71794579a385c250eaa5b68d" member_type: VOTER last_known_addr { host: "127.14.147.194" port: 44675 } }
I20260929 22:54:09.067382 1554 raft_consensus.cc:374] T 1d1f4f04dca34d158952f455c4c947f4 P 450311e81b584aec8d79736ebd48a766 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 22:54:09.067440 1554 raft_consensus.cc:729] T 1d1f4f04dca34d158952f455c4c947f4 P 450311e81b584aec8d79736ebd48a766 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 450311e81b584aec8d79736ebd48a766, State: Initialized, Role: FOLLOWER
I20260929 22:54:09.067559 1554 consensus_queue.cc:260] T 1d1f4f04dca34d158952f455c4c947f4 P 450311e81b584aec8d79736ebd48a766 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "450311e81b584aec8d79736ebd48a766" member_type: VOTER last_known_addr { host: "127.14.147.195" port: 39383 } } peers { permanent_uuid: "8d3e3ac5cd67441aa4ae1aa81d910722" member_type: VOTER last_known_addr { host: "127.14.147.193" port: 40941 } } peers { permanent_uuid: "7ef432ec71794579a385c250eaa5b68d" member_type: VOTER last_known_addr { host: "127.14.147.194" port: 44675 } }
I20260929 22:54:09.067504 1556 raft_consensus.cc:348] T 1d1f4f04dca34d158952f455c4c947f4 P 7ef432ec71794579a385c250eaa5b68d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "450311e81b584aec8d79736ebd48a766" member_type: VOTER last_known_addr { host: "127.14.147.195" port: 39383 } } peers { permanent_uuid: "8d3e3ac5cd67441aa4ae1aa81d910722" member_type: VOTER last_known_addr { host: "127.14.147.193" port: 40941 } } peers { permanent_uuid: "7ef432ec71794579a385c250eaa5b68d" member_type: VOTER last_known_addr { host: "127.14.147.194" port: 44675 } }
I20260929 22:54:09.067653 1556 raft_consensus.cc:374] T 1d1f4f04dca34d158952f455c4c947f4 P 7ef432ec71794579a385c250eaa5b68d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 22:54:09.067689 1556 raft_consensus.cc:729] T 1d1f4f04dca34d158952f455c4c947f4 P 7ef432ec71794579a385c250eaa5b68d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7ef432ec71794579a385c250eaa5b68d, State: Initialized, Role: FOLLOWER
I20260929 22:54:09.067811 1554 ts_tablet_manager.cc:1428] T 1d1f4f04dca34d158952f455c4c947f4 P 450311e81b584aec8d79736ebd48a766: Time spent starting tablet: real 0.003s user 0.002s sys 0.001s
I20260929 22:54:09.067790 1556 consensus_queue.cc:260] T 1d1f4f04dca34d158952f455c4c947f4 P 7ef432ec71794579a385c250eaa5b68d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "450311e81b584aec8d79736ebd48a766" member_type: VOTER last_known_addr { host: "127.14.147.195" port: 39383 } } peers { permanent_uuid: "8d3e3ac5cd67441aa4ae1aa81d910722" member_type: VOTER last_known_addr { host: "127.14.147.193" port: 40941 } } peers { permanent_uuid: "7ef432ec71794579a385c250eaa5b68d" member_type: VOTER last_known_addr { host: "127.14.147.194" port: 44675 } }
I20260929 22:54:09.067865 1540 heartbeater.cc:503] Master 127.14.147.254:35717 was elected leader, sending a full tablet report...
I20260929 22:54:09.068034 1556 ts_tablet_manager.cc:1428] T 1d1f4f04dca34d158952f455c4c947f4 P 7ef432ec71794579a385c250eaa5b68d: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260929 22:54:09.068122 1409 heartbeater.cc:503] Master 127.14.147.254:35717 was elected leader, sending a full tablet report...
I20260929 22:54:09.068468 1555 tablet_bootstrap.cc:492] T 1d1f4f04dca34d158952f455c4c947f4 P 8d3e3ac5cd67441aa4ae1aa81d910722: No bootstrap required, opened a new log
I20260929 22:54:09.068542 1555 ts_tablet_manager.cc:1397] T 1d1f4f04dca34d158952f455c4c947f4 P 8d3e3ac5cd67441aa4ae1aa81d910722: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.000s
I20260929 22:54:09.070559 1555 raft_consensus.cc:348] T 1d1f4f04dca34d158952f455c4c947f4 P 8d3e3ac5cd67441aa4ae1aa81d910722 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "450311e81b584aec8d79736ebd48a766" member_type: VOTER last_known_addr { host: "127.14.147.195" port: 39383 } } peers { permanent_uuid: "8d3e3ac5cd67441aa4ae1aa81d910722" member_type: VOTER last_known_addr { host: "127.14.147.193" port: 40941 } } peers { permanent_uuid: "7ef432ec71794579a385c250eaa5b68d" member_type: VOTER last_known_addr { host: "127.14.147.194" port: 44675 } }
I20260929 22:54:09.070716 1555 raft_consensus.cc:374] T 1d1f4f04dca34d158952f455c4c947f4 P 8d3e3ac5cd67441aa4ae1aa81d910722 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 22:54:09.070748 1555 raft_consensus.cc:729] T 1d1f4f04dca34d158952f455c4c947f4 P 8d3e3ac5cd67441aa4ae1aa81d910722 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8d3e3ac5cd67441aa4ae1aa81d910722, State: Initialized, Role: FOLLOWER
I20260929 22:54:09.070860 1555 consensus_queue.cc:260] T 1d1f4f04dca34d158952f455c4c947f4 P 8d3e3ac5cd67441aa4ae1aa81d910722 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "450311e81b584aec8d79736ebd48a766" member_type: VOTER last_known_addr { host: "127.14.147.195" port: 39383 } } peers { permanent_uuid: "8d3e3ac5cd67441aa4ae1aa81d910722" member_type: VOTER last_known_addr { host: "127.14.147.193" port: 40941 } } peers { permanent_uuid: "7ef432ec71794579a385c250eaa5b68d" member_type: VOTER last_known_addr { host: "127.14.147.194" port: 44675 } }
I20260929 22:54:09.071082 1555 ts_tablet_manager.cc:1428] T 1d1f4f04dca34d158952f455c4c947f4 P 8d3e3ac5cd67441aa4ae1aa81d910722: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260929 22:54:09.071142 1278 heartbeater.cc:503] Master 127.14.147.254:35717 was elected leader, sending a full tablet report...
I20260929 22:54:09.129117 1560 raft_consensus.cc:482] T 1d1f4f04dca34d158952f455c4c947f4 P 450311e81b584aec8d79736ebd48a766 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 22:54:09.129288 1560 raft_consensus.cc:504] T 1d1f4f04dca34d158952f455c4c947f4 P 450311e81b584aec8d79736ebd48a766 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "450311e81b584aec8d79736ebd48a766" member_type: VOTER last_known_addr { host: "127.14.147.195" port: 39383 } } peers { permanent_uuid: "8d3e3ac5cd67441aa4ae1aa81d910722" member_type: VOTER last_known_addr { host: "127.14.147.193" port: 40941 } } peers { permanent_uuid: "7ef432ec71794579a385c250eaa5b68d" member_type: VOTER last_known_addr { host: "127.14.147.194" port: 44675 } }
I20260929 22:54:09.129680 1560 leader_election.cc:288] T 1d1f4f04dca34d158952f455c4c947f4 P 450311e81b584aec8d79736ebd48a766 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8d3e3ac5cd67441aa4ae1aa81d910722 (127.14.147.193:40941), 7ef432ec71794579a385c250eaa5b68d (127.14.147.194:44675)
I20260929 22:54:09.133855 1551 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:54:09.129908 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.147.195:36347 (local address 127.14.147.193:40941)
0929 22:54:09.129968 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0929 22:54:09.129970 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 22:54:09.129980 (+ 10us) server_negotiation.cc:407] Connection header received
0929 22:54:09.130064 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 22:54:09.130065 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 22:54:09.130077 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 22:54:09.130141 (+ 64us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 22:54:09.130302 (+ 161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:54:09.131063 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:54:09.132892 (+ 1829us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:54:09.133473 (+ 581us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:54:09.133556 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 22:54:09.133565 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0929 22:54:09.133731 (+ 166us) server_negotiation.cc:299] Negotiation successful
0929 22:54:09.133772 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260929 22:54:09.133852 1552 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:54:09.130005 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.147.195:46857 (local address 127.14.147.194:44675)
0929 22:54:09.130060 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0929 22:54:09.130062 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 22:54:09.130099 (+ 37us) server_negotiation.cc:407] Connection header received
0929 22:54:09.130163 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 22:54:09.130164 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 22:54:09.130172 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 22:54:09.130217 (+ 45us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 22:54:09.130341 (+ 124us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:54:09.131060 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:54:09.132892 (+ 1832us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:54:09.133476 (+ 584us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:54:09.133556 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 22:54:09.133565 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0929 22:54:09.133731 (+ 166us) server_negotiation.cc:299] Negotiation successful
0929 22:54:09.133772 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260929 22:54:09.133879 1542 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:54:09.129851 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.147.193:40941 (local address 127.14.147.195:36347)
0929 22:54:09.129908 (+ 57us) negotiation.cc:107] Waiting for socket to connect
0929 22:54:09.129929 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0929 22:54:09.129975 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 22:54:09.130151 (+ 176us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 22:54:09.130152 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 22:54:09.130154 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 22:54:09.130228 (+ 74us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 22:54:09.130229 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 22:54:09.131251 (+ 1022us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 22:54:09.131254 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 22:54:09.132789 (+ 1535us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 22:54:09.132796 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 22:54:09.133565 (+ 769us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 22:54:09.133566 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 22:54:09.133656 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 22:54:09.133663 (+ 7us) client_negotiation.cc:769] Sending connection context
0929 22:54:09.133719 (+ 56us) client_negotiation.cc:240] Negotiation successful
0929 22:54:09.133780 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":37,"mutex_wait_us":58}
I20260929 22:54:09.133893 1563 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:54:09.129881 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.147.194:44675 (local address 127.14.147.195:46857)
0929 22:54:09.130051 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0929 22:54:09.130060 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0929 22:54:09.130104 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 22:54:09.130227 (+ 123us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 22:54:09.130227 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 22:54:09.130229 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 22:54:09.130280 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 22:54:09.130281 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 22:54:09.131251 (+ 970us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 22:54:09.131254 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 22:54:09.132802 (+ 1548us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 22:54:09.132803 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 22:54:09.133566 (+ 763us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 22:54:09.133566 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 22:54:09.133656 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 22:54:09.133663 (+ 7us) client_negotiation.cc:769] Sending connection context
0929 22:54:09.133719 (+ 56us) client_negotiation.cc:240] Negotiation successful
0929 22:54:09.133779 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":155,"mutex_wait_us":40,"thread_start_us":104,"threads_started":1}
I20260929 22:54:09.134265 1232 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1d1f4f04dca34d158952f455c4c947f4" candidate_uuid: "450311e81b584aec8d79736ebd48a766" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8d3e3ac5cd67441aa4ae1aa81d910722" is_pre_election: true
I20260929 22:54:09.134457 1232 raft_consensus.cc:2454] T 1d1f4f04dca34d158952f455c4c947f4 P 8d3e3ac5cd67441aa4ae1aa81d910722 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 450311e81b584aec8d79736ebd48a766 in term 0.
I20260929 22:54:09.134438 1363 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1d1f4f04dca34d158952f455c4c947f4" candidate_uuid: "450311e81b584aec8d79736ebd48a766" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7ef432ec71794579a385c250eaa5b68d" is_pre_election: true
I20260929 22:54:09.134572 1363 raft_consensus.cc:2454] T 1d1f4f04dca34d158952f455c4c947f4 P 7ef432ec71794579a385c250eaa5b68d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 450311e81b584aec8d79736ebd48a766 in term 0.
I20260929 22:54:09.134720 1430 leader_election.cc:302] T 1d1f4f04dca34d158952f455c4c947f4 P 450311e81b584aec8d79736ebd48a766 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 450311e81b584aec8d79736ebd48a766, 8d3e3ac5cd67441aa4ae1aa81d910722; no voters:
I20260929 22:54:09.134865 1560 raft_consensus.cc:2781] T 1d1f4f04dca34d158952f455c4c947f4 P 450311e81b584aec8d79736ebd48a766 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 22:54:09.134922 1560 raft_consensus.cc:482] T 1d1f4f04dca34d158952f455c4c947f4 P 450311e81b584aec8d79736ebd48a766 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 22:54:09.134943 1560 raft_consensus.cc:3037] T 1d1f4f04dca34d158952f455c4c947f4 P 450311e81b584aec8d79736ebd48a766 [term 0 FOLLOWER]: Advancing to term 1
I20260929 22:54:09.135664 1560 raft_consensus.cc:504] T 1d1f4f04dca34d158952f455c4c947f4 P 450311e81b584aec8d79736ebd48a766 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "450311e81b584aec8d79736ebd48a766" member_type: VOTER last_known_addr { host: "127.14.147.195" port: 39383 } } peers { permanent_uuid: "8d3e3ac5cd67441aa4ae1aa81d910722" member_type: VOTER last_known_addr { host: "127.14.147.193" port: 40941 } } peers { permanent_uuid: "7ef432ec71794579a385c250eaa5b68d" member_type: VOTER last_known_addr { host: "127.14.147.194" port: 44675 } }
I20260929 22:54:09.135793 1560 leader_election.cc:288] T 1d1f4f04dca34d158952f455c4c947f4 P 450311e81b584aec8d79736ebd48a766 [CANDIDATE]: Term 1 election: Requested vote from peers 8d3e3ac5cd67441aa4ae1aa81d910722 (127.14.147.193:40941), 7ef432ec71794579a385c250eaa5b68d (127.14.147.194:44675)
I20260929 22:54:09.135967 1363 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1d1f4f04dca34d158952f455c4c947f4" candidate_uuid: "450311e81b584aec8d79736ebd48a766" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7ef432ec71794579a385c250eaa5b68d"
I20260929 22:54:09.135969 1232 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1d1f4f04dca34d158952f455c4c947f4" candidate_uuid: "450311e81b584aec8d79736ebd48a766" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8d3e3ac5cd67441aa4ae1aa81d910722"
I20260929 22:54:09.136050 1363 raft_consensus.cc:3037] T 1d1f4f04dca34d158952f455c4c947f4 P 7ef432ec71794579a385c250eaa5b68d [term 0 FOLLOWER]: Advancing to term 1
I20260929 22:54:09.136050 1232 raft_consensus.cc:3037] T 1d1f4f04dca34d158952f455c4c947f4 P 8d3e3ac5cd67441aa4ae1aa81d910722 [term 0 FOLLOWER]: Advancing to term 1
I20260929 22:54:09.136955 1232 raft_consensus.cc:2454] T 1d1f4f04dca34d158952f455c4c947f4 P 8d3e3ac5cd67441aa4ae1aa81d910722 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 450311e81b584aec8d79736ebd48a766 in term 1.
I20260929 22:54:09.136955 1363 raft_consensus.cc:2454] T 1d1f4f04dca34d158952f455c4c947f4 P 7ef432ec71794579a385c250eaa5b68d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 450311e81b584aec8d79736ebd48a766 in term 1.
I20260929 22:54:09.137156 1428 leader_election.cc:302] T 1d1f4f04dca34d158952f455c4c947f4 P 450311e81b584aec8d79736ebd48a766 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 450311e81b584aec8d79736ebd48a766, 7ef432ec71794579a385c250eaa5b68d; no voters:
I20260929 22:54:09.137262 1560 raft_consensus.cc:2781] T 1d1f4f04dca34d158952f455c4c947f4 P 450311e81b584aec8d79736ebd48a766 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 22:54:09.137398 1560 raft_consensus.cc:686] T 1d1f4f04dca34d158952f455c4c947f4 P 450311e81b584aec8d79736ebd48a766 [term 1 LEADER]: Becoming Leader. State: Replica: 450311e81b584aec8d79736ebd48a766, State: Running, Role: LEADER
I20260929 22:54:09.137491 1560 consensus_queue.cc:237] T 1d1f4f04dca34d158952f455c4c947f4 P 450311e81b584aec8d79736ebd48a766 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "450311e81b584aec8d79736ebd48a766" member_type: VOTER last_known_addr { host: "127.14.147.195" port: 39383 } } peers { permanent_uuid: "8d3e3ac5cd67441aa4ae1aa81d910722" member_type: VOTER last_known_addr { host: "127.14.147.193" port: 40941 } } peers { permanent_uuid: "7ef432ec71794579a385c250eaa5b68d" member_type: VOTER last_known_addr { host: "127.14.147.194" port: 44675 } }
I20260929 22:54:09.138301 1089 catalog_manager.cc:5495] T 1d1f4f04dca34d158952f455c4c947f4 P 450311e81b584aec8d79736ebd48a766 reported cstate change: term changed from 0 to 1, leader changed from <none> to 450311e81b584aec8d79736ebd48a766 (127.14.147.195). New cstate: current_term: 1 leader_uuid: "450311e81b584aec8d79736ebd48a766" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "450311e81b584aec8d79736ebd48a766" member_type: VOTER last_known_addr { host: "127.14.147.195" port: 39383 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8d3e3ac5cd67441aa4ae1aa81d910722" member_type: VOTER last_known_addr { host: "127.14.147.193" port: 40941 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7ef432ec71794579a385c250eaa5b68d" member_type: VOTER last_known_addr { host: "127.14.147.194" port: 44675 } health_report { overall_health: UNKNOWN } } }
I20260929 22:54:09.155300 1553 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:54:09.153146 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36796 (local address 127.14.147.195:39383)
0929 22:54:09.153195 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0929 22:54:09.153196 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0929 22:54:09.153203 (+ 7us) server_negotiation.cc:407] Connection header received
0929 22:54:09.153239 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 22:54:09.153239 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 22:54:09.153260 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 22:54:09.153318 (+ 58us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 22:54:09.153430 (+ 112us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:54:09.153899 (+ 469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:54:09.154650 (+ 751us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:54:09.154848 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:54:09.154905 (+ 57us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 22:54:09.155009 (+ 104us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 22:54:09.155097 (+ 88us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 22:54:09.155163 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0929 22:54:09.155183 (+ 20us) server_negotiation.cc:299] Negotiation successful
0929 22:54:09.155190 (+ 7us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260929 22:54:09.157052 1232 raft_consensus.cc:1261] T 1d1f4f04dca34d158952f455c4c947f4 P 8d3e3ac5cd67441aa4ae1aa81d910722 [term 1 FOLLOWER]: Refusing update from remote peer 450311e81b584aec8d79736ebd48a766: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 22:54:09.157051 1363 raft_consensus.cc:1261] T 1d1f4f04dca34d158952f455c4c947f4 P 7ef432ec71794579a385c250eaa5b68d [term 1 FOLLOWER]: Refusing update from remote peer 450311e81b584aec8d79736ebd48a766: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 22:54:09.157405 1564 consensus_queue.cc:1035] T 1d1f4f04dca34d158952f455c4c947f4 P 450311e81b584aec8d79736ebd48a766 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8d3e3ac5cd67441aa4ae1aa81d910722" member_type: VOTER last_known_addr { host: "127.14.147.193" port: 40941 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 22:54:09.157544 1560 consensus_queue.cc:1035] T 1d1f4f04dca34d158952f455c4c947f4 P 450311e81b584aec8d79736ebd48a766 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7ef432ec71794579a385c250eaa5b68d" member_type: VOTER last_known_addr { host: "127.14.147.194" port: 44675 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 22:54:09.159961 1567 mvcc.cc:205] Tried to move back new op lower bound from 7334799151744835584 to 7334799151667859456. Current Snapshot: MvccSnapshot[applied={T|T < 7334799151744835584}]
I20260929 22:54:09.160889 1569 mvcc.cc:205] Tried to move back new op lower bound from 7334799151744835584 to 7334799151667859456. Current Snapshot: MvccSnapshot[applied={T|T < 7334799151744835584}]
I20260929 22:54:09.160998 1568 mvcc.cc:205] Tried to move back new op lower bound from 7334799151744835584 to 7334799151667859456. Current Snapshot: MvccSnapshot[applied={T|T < 7334799151744835584 or (T in {7334799151744835584})}]
I20260929 22:54:09.163156 1089 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:54958:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 22:54:09.163256 1089 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:54958:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 22:54:09.167657 1089 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 4a2790239aea42bc845c0eb52f6c83a7: Sending DeleteTablet for 3 replicas of tablet 1d1f4f04dca34d158952f455c4c947f4
I20260929 22:54:09.168152 1212 tablet_service.cc:1514] Processing DeleteTablet for tablet 1d1f4f04dca34d158952f455c4c947f4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 22:54:09 UTC) from {username='slave'} at 127.0.0.1:51470
I20260929 22:54:09.168131 1474 tablet_service.cc:1514] Processing DeleteTablet for tablet 1d1f4f04dca34d158952f455c4c947f4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 22:54:09 UTC) from {username='slave'} at 127.0.0.1:36792
I20260929 22:54:09.168589 1343 tablet_service.cc:1514] Processing DeleteTablet for tablet 1d1f4f04dca34d158952f455c4c947f4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 22:54:09 UTC) from {username='slave'} at 127.0.0.1:50308
I20260929 22:54:09.168824 1579 tablet_replica.cc:330] T 1d1f4f04dca34d158952f455c4c947f4 P 7ef432ec71794579a385c250eaa5b68d: stopping tablet replica
I20260929 22:54:09.168936 1579 raft_consensus.cc:2229] T 1d1f4f04dca34d158952f455c4c947f4 P 7ef432ec71794579a385c250eaa5b68d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 22:54:09.169049 1579 raft_consensus.cc:2258] T 1d1f4f04dca34d158952f455c4c947f4 P 7ef432ec71794579a385c250eaa5b68d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 22:54:09.169181 14927 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhWOCLk/build/release/bin/kudu with pid 1150
I20260929 22:54:09.170049 1577 tablet_replica.cc:330] T 1d1f4f04dca34d158952f455c4c947f4 P 450311e81b584aec8d79736ebd48a766: stopping tablet replica
I20260929 22:54:09.170161 1577 raft_consensus.cc:2229] T 1d1f4f04dca34d158952f455c4c947f4 P 450311e81b584aec8d79736ebd48a766 [term 1 LEADER]: Raft consensus shutting down.
I20260929 22:54:09.170259 1577 raft_consensus.cc:2258] T 1d1f4f04dca34d158952f455c4c947f4 P 450311e81b584aec8d79736ebd48a766 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 22:54:09.172133 1579 ts_tablet_manager.cc:1905] T 1d1f4f04dca34d158952f455c4c947f4 P 7ef432ec71794579a385c250eaa5b68d: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260929 22:54:09.175525 1077 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.14.147.193:40941 (error 108)
I20260929 22:54:09.176133 1579 ts_tablet_manager.cc:1918] T 1d1f4f04dca34d158952f455c4c947f4 P 7ef432ec71794579a385c250eaa5b68d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 22:54:09.176218 1579 log.cc:1196] T 1d1f4f04dca34d158952f455c4c947f4 P 7ef432ec71794579a385c250eaa5b68d: Deleting WAL directory at /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790722213368738-14927-0/minicluster-data/ts-1/wal/wals/1d1f4f04dca34d158952f455c4c947f4
W20260929 22:54:09.176262 1550 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 22:54:09.175953 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.147.193:40941 (local address 127.0.0.1:51474)
0929 22:54:09.176015 (+ 62us) negotiation.cc:107] Waiting for socket to connect
0929 22:54:09.176027 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0929 22:54:09.176069 (+ 42us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 22:54:09.176183 (+ 114us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.147.193:40941: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":41}
I20260929 22:54:09.176463 1579 ts_tablet_manager.cc:1939] T 1d1f4f04dca34d158952f455c4c947f4 P 7ef432ec71794579a385c250eaa5b68d: Deleting consensus metadata
W20260929 22:54:09.176800 1077 catalog_manager.cc:4584] TS 8d3e3ac5cd67441aa4ae1aa81d910722 (127.14.147.193:40941): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 1d1f4f04dca34d158952f455c4c947f4: Network error: Client connection negotiation failed: client connection to 127.14.147.193:40941: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260929 22:54:09.176952 1075 catalog_manager.cc:4841] TS 7ef432ec71794579a385c250eaa5b68d (127.14.147.194:44675): tablet 1d1f4f04dca34d158952f455c4c947f4 (table test-table [id=5e9ea8225a7a4d2492246854be2a539e]) successfully deleted
I20260929 22:54:09.177363 14927 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhWOCLk/build/release/bin/kudu with pid 1281
I20260929 22:54:09.180420 1577 ts_tablet_manager.cc:1905] T 1d1f4f04dca34d158952f455c4c947f4 P 450311e81b584aec8d79736ebd48a766: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 22:54:09.182112 1577 ts_tablet_manager.cc:1918] T 1d1f4f04dca34d158952f455c4c947f4 P 450311e81b584aec8d79736ebd48a766: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 22:54:09.182209 1577 log.cc:1196] T 1d1f4f04dca34d158952f455c4c947f4 P 450311e81b584aec8d79736ebd48a766: Deleting WAL directory at /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790722213368738-14927-0/minicluster-data/ts-2/wal/wals/1d1f4f04dca34d158952f455c4c947f4
I20260929 22:54:09.182471 1577 ts_tablet_manager.cc:1939] T 1d1f4f04dca34d158952f455c4c947f4 P 450311e81b584aec8d79736ebd48a766: Deleting consensus metadata
I20260929 22:54:09.183252 1076 catalog_manager.cc:4841] TS 450311e81b584aec8d79736ebd48a766 (127.14.147.195:39383): tablet 1d1f4f04dca34d158952f455c4c947f4 (table test-table [id=5e9ea8225a7a4d2492246854be2a539e]) successfully deleted
I20260929 22:54:09.183861 14927 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhWOCLk/build/release/bin/kudu with pid 1412
I20260929 22:54:09.189361 14927 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhWOCLk/build/release/bin/kudu with pid 1059
2026-09-29T22:54:09Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/3 (702 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/4
Loading random data
Initializing database '/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790722213368738-14927-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 22:54:09 dist-test-slave-98mk krb5kdc[1583](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 22:54:09 dist-test-slave-98mk krb5kdc[1583](info): set up 2 sockets
Sep 29 22:54:09 dist-test-slave-98mk krb5kdc[1583](info): commencing operation
krb5kdc: starting...
W20260929 22:54:10.155876 14927 mini_kdc.cc:121] Time spent starting KDC: real 0.944s user 0.003s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790722213368738-14927-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790722213368738-14927-0/krb5kdc/test-user.keytab.
Sep 29 22:54:10 dist-test-slave-98mk krb5kdc[1583](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790722450, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T22:54:10Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T22:54:10Z Disabled control of system clock
WARNING: no policy specified for kudu/127.14.147.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.147.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.147.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790722213368738-14927-0/krb5kdc/kudu_127.14.147.254.keytab.
Entry for principal kudu/127.14.147.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790722213368738-14927-0/krb5kdc/kudu_127.14.147.254.keytab.
I20260929 22:54:10.268121 14927 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhWOCLk/build/release/bin/kudu
/tmp/dist-test-taskhWOCLk/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790722213368738-14927-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790722213368738-14927-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790722213368738-14927-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790722213368738-14927-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.147.254:33425
--webserver_interface=127.14.147.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790722213368738-14927-0/krb5kdc/kudu_127.14.147.254.keytab
--principal=kudu/127.14.147.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.147.212:46383
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.147.254:33425
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790722213368738-14927-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790722213368738-14927-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790722213368738-14927-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 22:54:10.355105 1596 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 22:54:10.355346 1596 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 22:54:10.355373 1596 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 22:54:10.356667 1596 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 22:54:10.356720 1596 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 22:54:10.356734 1596 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 22:54:10.356745 1596 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 22:54:10.356756 1596 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 22:54:10.358355 1596 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.147.212:46383
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790722213368738-14927-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790722213368738-14927-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.14.147.254:33425
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790722213368738-14927-0/krb5kdc/kudu_127.14.147.254.keytab
--principal=kudu/127.14.147.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.147.254:33425
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790722213368738-14927-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.14.147.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1596
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790722213368738-14927-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 21:58:54 UTC on ccycloud.quasar-xguoti.root.comops.site
build id 4681390
I20260929 22:54:10.358613 1596 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 22:54:10.358840 1596 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260929 22:54:10.363665 1596 server_base.cc:1034] running on GCE node
W20260929 22:54:10.363623 1604 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 22:54:10.363621 1601 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 22:54:10.363621 1602 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 22:54:10.364112 1596 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 22:54:10.364421 1596 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 22:54:10.365674 1596 hybrid_clock.cc:648] HybridClock initialized: now 1790722450365658 us; error 50 us; skew 500 ppm
Sep 29 22:54:10 dist-test-slave-98mk krb5kdc[1583](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790722450, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.147.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 22:54:10.368852 1596 init.cc:377] Logged in from keytab as kudu/127.14.147.254@KRBTEST.COM (short username kudu)
I20260929 22:54:10.369352 1596 webserver.cc:460] Webserver started at http://127.14.147.254:38905/ using document root <none> and password file <none>
I20260929 22:54:10.369580 1596 fs_manager.cc:359] Metadata directory not provided
I20260929 22:54:10.369626 1596 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 22:54:10.369766 1596 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 22:54:10.370846 1596 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790722213368738-14927-0/minicluster-data/master-0/data/instance:
uuid: "e51dbed0f1914a609f10ad8a571d5cec"
format_stamp: "Formatted at 2026-09-29 22:54:10 on dist-test-slave-98mk"
server_key: "58550cb6b18e5f53d8c6a85e34f72d19"
server_key_iv: "e1b8a072e225f728f47268d208c4ed40"
server_key_version: "encryptionkey@0"
I20260929 22:54:10.371208 1596 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790722213368738-14927-0/minicluster-data/master-0/wal/instance:
uuid: "e51dbed0f1914a609f10ad8a571d5cec"
format_stamp: "Formatted at 2026-09-29 22:54:10 on dist-test-slave-98mk"
server_key: "58550cb6b18e5f53d8c6a85e34f72d19"
server_key_iv: "e1b8a072e225f728f47268d208c4ed40"
server_key_version: "encryptionkey@0"
I20260929 22:54:10.372625 1596 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260929 22:54:10.373344 1611 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 22:54:10.373545 1596 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260929 22:54:10.373629 1596 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790722213368738-14927-0/minicluster-data/master-0/data,/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790722213368738-14927-0/minicluster-data/master-0/wal
uuid: "e51dbed0f1914a609f10ad8a571d5cec"
format_stamp: "Formatted at 2026-09-29 22:54:10 on dist-test-slave-98mk"
server_key: "58550cb6b18e5f53d8c6a85e34f72d19"
server_key_iv: "e1b8a072e225f728f47268d208c4ed40"
server_key_version: "encryptionkey@0"
I20260929 22:54:10.373703 1596 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790722213368738-14927-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790722213368738-14927-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790722213368738-14927-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 22:54:10.381345 1596 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 22:54:10.383248 1596 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 22:54:10.383386 1596 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 22:54:10.387526 1596 rpc_server.cc:307] RPC server started. Bound to: 127.14.147.254:33425
I20260929 22:54:10.387573 1663 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.147.254:33425 every 8 connection(s)
I20260929 22:54:10.387929 1596 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790722213368738-14927-0/minicluster-data/master-0/data/info.pb
I20260929 22:54:10.388772 1664 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 22:54:10.391142 1664 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e51dbed0f1914a609f10ad8a571d5cec: Bootstrap starting.
I20260929 22:54:10.391782 1664 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e51dbed0f1914a609f10ad8a571d5cec: Neither blocks nor log segments found. Creating new log.
I20260929 22:54:10.392063 1664 log.cc:824] T 00000000000000000000000000000000 P e51dbed0f1914a609f10ad8a571d5cec: Log is configured to *not* fsync() on all Append() calls
I20260929 22:54:10.392791 1664 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e51dbed0f1914a609f10ad8a571d5cec: No bootstrap required, opened a new log
I20260929 22:54:10.393893 14927 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhWOCLk/build/release/bin/kudu as pid 1596
I20260929 22:54:10.394017 14927 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790722213368738-14927-0/minicluster-data/master-0/wal/instance
I20260929 22:54:10.394213 14927 external_mini_cluster.cc:1428] Setting key 727f269c9ba47579f2ec82741edd0733
I20260929 22:54:10.394743 1664 raft_consensus.cc:348] T 00000000000000000000000000000000 P e51dbed0f1914a609f10ad8a571d5cec [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e51dbed0f1914a609f10ad8a571d5cec" member_type: VOTER last_known_addr { host: "127.14.147.254" port: 33425 } }
I20260929 22:54:10.394973 1664 raft_consensus.cc:374] T 00000000000000000000000000000000 P e51dbed0f1914a609f10ad8a571d5cec [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 22:54:10.395027 1664 raft_consensus.cc:729] T 00000000000000000000000000000000 P e51dbed0f1914a609f10ad8a571d5cec [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e51dbed0f1914a609f10ad8a571d5cec, State: Initialized, Role: FOLLOWER
I20260929 22:54:10.395160 1664 consensus_queue.cc:260] T 00000000000000000000000000000000 P e51dbed0f1914a609f10ad8a571d5cec [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e51dbed0f1914a609f10ad8a571d5cec" member_type: VOTER last_known_addr { host: "127.14.147.254" port: 33425 } }
I20260929 22:54:10.395239 1664 raft_consensus.cc:388] T 00000000000000000000000000000000 P e51dbed0f1914a609f10ad8a571d5cec [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 22:54:10.395272 1664 raft_consensus.cc:482] T 00000000000000000000000000000000 P e51dbed0f1914a609f10ad8a571d5cec [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 22:54:10.395314 1664 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e51dbed0f1914a609f10ad8a571d5cec [term 0 FOLLOWER]: Advancing to term 1
I20260929 22:54:10.396277 1664 raft_consensus.cc:504] T 00000000000000000000000000000000 P e51dbed0f1914a609f10ad8a571d5cec [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e51dbed0f1914a609f10ad8a571d5cec" member_type: VOTER last_known_addr { host: "127.14.147.254" port: 33425 } }
I20260929 22:54:10.396382 1664 leader_election.cc:302] T 00000000000000000000000000000000 P e51dbed0f1914a609f10ad8a571d5cec [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e51dbed0f1914a609f10ad8a571d5cec; no voters:
I20260929 22:54:10.396544 1664 leader_election.cc:288] T 00000000000000000000000000000000 P e51dbed0f1914a609f10ad8a571d5cec [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 22:54:10.396588 1669 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e51dbed0f1914a609f10ad8a571d5cec [term 1 FOLLOWER]: Leader election won for term 1
I20260929 22:54:10.396699 1669 raft_consensus.cc:686] T 00000000000000000000000000000000 P e51dbed0f1914a609f10ad8a571d5cec [term 1 LEADER]: Becoming Leader. State: Replica: e51dbed0f1914a609f10ad8a571d5cec, State: Running, Role: LEADER
I20260929 22:54:10.396785 1669 consensus_queue.cc:237] T 00000000000000000000000000000000 P e51dbed0f1914a609f10ad8a571d5cec [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e51dbed0f1914a609f10ad8a571d5cec" member_type: VOTER last_known_addr { host: "127.14.147.254" port: 33425 } }
I20260929 22:54:10.396834 1664 sys_catalog.cc:564] T 00000000000000000000000000000000 P e51dbed0f1914a609f10ad8a571d5cec [sys.catalog]: configured and running, proceeding with master startup.
I20260929 22:54:10.397250 1671 sys_catalog.cc:455] T 00000000000000000000000000000000 P e51dbed0f1914a609f10ad8a571d5cec [sys.catalog]: SysCatalogTable state changed. Reason: New leader e51dbed0f1914a609f10ad8a571d5cec. Latest consensus state: current_term: 1 leader_uuid: "e51dbed0f1914a609f10ad8a571d5cec" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e51dbed0f1914a609f10ad8a571d5cec" member_type: VOTER last_known_addr { host: "127.14.147.254" port: 33425 } } }
I20260929 22:54:10.397259 1670 sys_catalog.cc:455] T 00000000000000000000000000000000 P e51dbed0f1914a609f10ad8a571d5cec [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e51dbed0f1914a609f10ad8a571d5cec" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e51dbed0f1914a609f10ad8a571d5cec" member_type: VOTER last_known_addr { host: "127.14.147.254" port: 33425 } } }
I20260929 22:54:10.397356 1671 sys_catalog.cc:458] T 00000000000000000000000000000000 P e51dbed0f1914a609f10ad8a571d5cec [sys.catalog]: This master's current role is: LEADER
I20260929 22:54:10.397378 1670 sys_catalog.cc:458] T 00000000000000000000000000000000 P e51dbed0f1914a609f10ad8a571d5cec [sys.catalog]: This master's current role is: LEADER
W20260929 22:54:10.398664 1684 catalog_manager.cc:1558] T 00000000000000000000000000000000 P e51dbed0f1914a609f10ad8a571d5cec: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 22:54:10.398720 1684 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260929 22:54:10.398770 1677 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 22:54:10.398921 1677 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 22:54:10.401845 1677 catalog_manager.cc:1347] Generated new cluster ID: cc8a84bea630412abf3258c4a7740cd9
I20260929 22:54:10.401914 1677 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 29 22:54:10 dist-test-slave-98mk krb5kdc[1583](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790722450, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.147.254@KRBTEST.COM
I20260929 22:54:10.405727 1668 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:54:10.394964 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46238 (local address 127.14.147.254:33425)
0929 22:54:10.395245 (+ 281us) server_negotiation.cc:206] Beginning negotiation
0929 22:54:10.395256 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0929 22:54:10.395298 (+ 42us) server_negotiation.cc:407] Connection header received
0929 22:54:10.395535 (+ 237us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 22:54:10.395544 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 22:54:10.395607 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 22:54:10.395755 (+ 148us) server_negotiation.cc:226] Negotiated authn=SASL
0929 22:54:10.396731 (+ 976us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:54:10.398366 (+ 1635us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:54:10.399259 (+ 893us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:54:10.399839 (+ 580us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:54:10.399950 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 22:54:10.402604 (+ 2654us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 22:54:10.402609 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 22:54:10.402609 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 22:54:10.402622 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0929 22:54:10.404259 (+ 1637us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 22:54:10.404603 (+ 344us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 22:54:10.404606 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 22:54:10.404607 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 22:54:10.404706 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 22:54:10.404886 (+ 180us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 22:54:10.404887 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 22:54:10.404887 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 22:54:10.405132 (+ 245us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 22:54:10.405272 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0929 22:54:10.405385 (+ 113us) server_negotiation.cc:299] Negotiation successful
0929 22:54:10.405528 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":222,"thread_start_us":105,"threads_started":1}
I20260929 22:54:10.419353 1677 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 22:54:10.419917 1677 catalog_manager.cc:1504] Loading token signing keys...
I20260929 22:54:10.425763 1677 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e51dbed0f1914a609f10ad8a571d5cec: Generated new TSK 0
I20260929 22:54:10.425988 1677 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.14.147.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.147.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.147.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790722213368738-14927-0/krb5kdc/kudu_127.14.147.193.keytab.
Entry for principal kudu/127.14.147.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790722213368738-14927-0/krb5kdc/kudu_127.14.147.193.keytab.
I20260929 22:54:10.459270 14927 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhWOCLk/build/release/bin/kudu
/tmp/dist-test-taskhWOCLk/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790722213368738-14927-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790722213368738-14927-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790722213368738-14927-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790722213368738-14927-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.147.193:0
--local_ip_for_outbound_sockets=127.14.147.193
--webserver_interface=127.14.147.193
--webserver_port=0
--tserver_master_addrs=127.14.147.254:33425
--keytab_file=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790722213368738-14927-0/krb5kdc/kudu_127.14.147.193.keytab
--principal=kudu/127.14.147.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.147.212:46383
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790722213368738-14927-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790722213368738-14927-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790722213368738-14927-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 22:54:10.542011 1690 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 22:54:10.542243 1690 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 22:54:10.542277 1690 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 22:54:10.543569 1690 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 22:54:10.543625 1690 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 22:54:10.543664 1690 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.147.193
I20260929 22:54:10.545243 1690 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.147.212:46383
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790722213368738-14927-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790722213368738-14927-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790722213368738-14927-0/krb5kdc/kudu_127.14.147.193.keytab
--principal=kudu/127.14.147.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.147.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790722213368738-14927-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.14.147.193
--webserver_port=0
--tserver_master_addrs=127.14.147.254:33425
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1690
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.147.193
--log_dir=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790722213368738-14927-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 21:58:54 UTC on ccycloud.quasar-xguoti.root.comops.site
build id 4681390
I20260929 22:54:10.545501 1690 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 22:54:10.545724 1690 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 22:54:10.550688 1695 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 22:54:10.550851 1690 server_base.cc:1034] running on GCE node
W20260929 22:54:10.550699 1698 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 22:54:10.550699 1696 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 22:54:10.551386 1690 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 22:54:10.551887 1690 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 22:54:10.553094 1690 hybrid_clock.cc:648] HybridClock initialized: now 1790722450553062 us; error 52 us; skew 500 ppm
Sep 29 22:54:10 dist-test-slave-98mk krb5kdc[1583](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790722450, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.147.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 22:54:10.555838 1690 init.cc:377] Logged in from keytab as kudu/127.14.147.193@KRBTEST.COM (short username kudu)
I20260929 22:54:10.556298 1690 webserver.cc:460] Webserver started at http://127.14.147.193:36645/ using document root <none> and password file <none>
I20260929 22:54:10.556506 1690 fs_manager.cc:359] Metadata directory not provided
I20260929 22:54:10.556545 1690 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 22:54:10.556641 1690 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 22:54:10.557662 1690 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790722213368738-14927-0/minicluster-data/ts-0/data/instance:
uuid: "b9ae095e7af044e981b5f50ce77481a5"
format_stamp: "Formatted at 2026-09-29 22:54:10 on dist-test-slave-98mk"
server_key: "f25c576aa76910b7ad5f54bf5f41f0fb"
server_key_iv: "8b7a7f7f50b1062dc62e2314d0cbd542"
server_key_version: "encryptionkey@0"
I20260929 22:54:10.557948 1690 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790722213368738-14927-0/minicluster-data/ts-0/wal/instance:
uuid: "b9ae095e7af044e981b5f50ce77481a5"
format_stamp: "Formatted at 2026-09-29 22:54:10 on dist-test-slave-98mk"
server_key: "f25c576aa76910b7ad5f54bf5f41f0fb"
server_key_iv: "8b7a7f7f50b1062dc62e2314d0cbd542"
server_key_version: "encryptionkey@0"
I20260929 22:54:10.559228 1690 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260929 22:54:10.559991 1705 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 22:54:10.560186 1690 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260929 22:54:10.560263 1690 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790722213368738-14927-0/minicluster-data/ts-0/data,/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790722213368738-14927-0/minicluster-data/ts-0/wal
uuid: "b9ae095e7af044e981b5f50ce77481a5"
format_stamp: "Formatted at 2026-09-29 22:54:10 on dist-test-slave-98mk"
server_key: "f25c576aa76910b7ad5f54bf5f41f0fb"
server_key_iv: "8b7a7f7f50b1062dc62e2314d0cbd542"
server_key_version: "encryptionkey@0"
I20260929 22:54:10.560340 1690 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790722213368738-14927-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790722213368738-14927-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790722213368738-14927-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 22:54:10.579849 1690 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 22:54:10.581657 1690 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 22:54:10.581794 1690 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 22:54:10.582013 1690 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 22:54:10.582299 1690 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 22:54:10.582332 1690 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 22:54:10.582362 1690 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 22:54:10.582379 1690 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 22:54:10.587981 1690 rpc_server.cc:307] RPC server started. Bound to: 127.14.147.193:38915
I20260929 22:54:10.588035 1819 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.147.193:38915 every 8 connection(s)
I20260929 22:54:10.588333 1690 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790722213368738-14927-0/minicluster-data/ts-0/data/info.pb
Sep 29 22:54:10 dist-test-slave-98mk krb5kdc[1583](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790722450, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.147.193@KRBTEST.COM for kudu/127.14.147.254@KRBTEST.COM
I20260929 22:54:10.594895 14927 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhWOCLk/build/release/bin/kudu as pid 1690
I20260929 22:54:10.595005 14927 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790722213368738-14927-0/minicluster-data/ts-0/wal/instance
I20260929 22:54:10.595213 14927 external_mini_cluster.cc:1428] Setting key d8767d408d433a9d87757e95756bdad1
I20260929 22:54:10.598270 1668 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:54:10.589028 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.147.193:44895 (local address 127.14.147.254:33425)
0929 22:54:10.589082 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0929 22:54:10.589084 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 22:54:10.589250 (+ 166us) server_negotiation.cc:407] Connection header received
0929 22:54:10.589750 (+ 500us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 22:54:10.589751 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 22:54:10.589764 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 22:54:10.589815 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0929 22:54:10.590747 (+ 932us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:54:10.591273 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:54:10.592315 (+ 1042us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:54:10.592529 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:54:10.592587 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 22:54:10.595081 (+ 2494us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 22:54:10.595082 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 22:54:10.595083 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 22:54:10.595092 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0929 22:54:10.596718 (+ 1626us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 22:54:10.597150 (+ 432us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 22:54:10.597152 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 22:54:10.597152 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 22:54:10.597211 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 22:54:10.597458 (+ 247us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 22:54:10.597459 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 22:54:10.597459 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 22:54:10.597599 (+ 140us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 22:54:10.597668 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0929 22:54:10.598110 (+ 442us) server_negotiation.cc:299] Negotiation successful
0929 22:54:10.598188 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260929 22:54:10.598574 1822 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:54:10.588987 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.147.254:33425 (local address 127.14.147.193:44895)
0929 22:54:10.589129 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0929 22:54:10.589169 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0929 22:54:10.589651 (+ 482us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 22:54:10.589892 (+ 241us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 22:54:10.589896 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 22:54:10.590450 (+ 554us) client_negotiation.cc:189] Negotiated authn=SASL
0929 22:54:10.590645 (+ 195us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 22:54:10.590649 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 22:54:10.591366 (+ 717us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 22:54:10.591367 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 22:54:10.592225 (+ 858us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 22:54:10.592228 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 22:54:10.592607 (+ 379us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 22:54:10.592608 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 22:54:10.592728 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 22:54:10.593386 (+ 658us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 22:54:10.593395 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0929 22:54:10.594959 (+ 1564us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 22:54:10.596794 (+ 1835us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 22:54:10.596802 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 22:54:10.596804 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 22:54:10.597086 (+ 282us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 22:54:10.597292 (+ 206us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 22:54:10.597292 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 22:54:10.597293 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 22:54:10.597404 (+ 111us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 22:54:10.597696 (+ 292us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 22:54:10.597732 (+ 36us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 22:54:10.597931 (+ 199us) client_negotiation.cc:769] Sending connection context
0929 22:54:10.598088 (+ 157us) client_negotiation.cc:240] Negotiation successful
0929 22:54:10.598297 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":101,"thread_start_us":66,"threads_started":1}
I20260929 22:54:10.599066 1820 heartbeater.cc:347] Connected to a master server at 127.14.147.254:33425
I20260929 22:54:10.599169 1820 heartbeater.cc:464] Registering TS with master...
I20260929 22:54:10.599357 1820 heartbeater.cc:511] Master 127.14.147.254:33425 requested a full tablet report, sending...
I20260929 22:54:10.599891 1628 ts_manager.cc:194] Registered new tserver with Master: b9ae095e7af044e981b5f50ce77481a5 (127.14.147.193:38915)
I20260929 22:54:10.600562 1628 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.147.193@KRBTEST.COM'} at 127.14.147.193:44895
WARNING: no policy specified for kudu/127.14.147.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.147.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.147.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790722213368738-14927-0/krb5kdc/kudu_127.14.147.194.keytab.
Entry for principal kudu/127.14.147.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790722213368738-14927-0/krb5kdc/kudu_127.14.147.194.keytab.
I20260929 22:54:10.620333 14927 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhWOCLk/build/release/bin/kudu
/tmp/dist-test-taskhWOCLk/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790722213368738-14927-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790722213368738-14927-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790722213368738-14927-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790722213368738-14927-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.147.194:0
--local_ip_for_outbound_sockets=127.14.147.194
--webserver_interface=127.14.147.194
--webserver_port=0
--tserver_master_addrs=127.14.147.254:33425
--keytab_file=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790722213368738-14927-0/krb5kdc/kudu_127.14.147.194.keytab
--principal=kudu/127.14.147.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.147.212:46383
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790722213368738-14927-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790722213368738-14927-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790722213368738-14927-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 22:54:10.700223 1825 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 22:54:10.700444 1825 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 22:54:10.700470 1825 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 22:54:10.701695 1825 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 22:54:10.701740 1825 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 22:54:10.701768 1825 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.147.194
I20260929 22:54:10.703254 1825 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.147.212:46383
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790722213368738-14927-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790722213368738-14927-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790722213368738-14927-0/krb5kdc/kudu_127.14.147.194.keytab
--principal=kudu/127.14.147.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.147.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790722213368738-14927-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.14.147.194
--webserver_port=0
--tserver_master_addrs=127.14.147.254:33425
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1825
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.147.194
--log_dir=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790722213368738-14927-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 21:58:54 UTC on ccycloud.quasar-xguoti.root.comops.site
build id 4681390
I20260929 22:54:10.703512 1825 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 22:54:10.703738 1825 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 22:54:10.708490 1830 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 22:54:10.708501 1831 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 22:54:10.708508 1833 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 22:54:10.708709 1825 server_base.cc:1034] running on GCE node
I20260929 22:54:10.708905 1825 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 22:54:10.709103 1825 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 22:54:10.710233 1825 hybrid_clock.cc:648] HybridClock initialized: now 1790722450710216 us; error 30 us; skew 500 ppm
Sep 29 22:54:10 dist-test-slave-98mk krb5kdc[1583](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790722450, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.147.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 22:54:10.712652 1825 init.cc:377] Logged in from keytab as kudu/127.14.147.194@KRBTEST.COM (short username kudu)
I20260929 22:54:10.713094 1825 webserver.cc:460] Webserver started at http://127.14.147.194:38275/ using document root <none> and password file <none>
I20260929 22:54:10.713291 1825 fs_manager.cc:359] Metadata directory not provided
I20260929 22:54:10.713335 1825 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 22:54:10.713467 1825 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 22:54:10.714521 1825 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790722213368738-14927-0/minicluster-data/ts-1/data/instance:
uuid: "8f1ac88655df49388d45f319334ef628"
format_stamp: "Formatted at 2026-09-29 22:54:10 on dist-test-slave-98mk"
server_key: "b4df135b603a7325d9c096c45b11c785"
server_key_iv: "efdc9ff3c16a1388d3193c3757832522"
server_key_version: "encryptionkey@0"
I20260929 22:54:10.714834 1825 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790722213368738-14927-0/minicluster-data/ts-1/wal/instance:
uuid: "8f1ac88655df49388d45f319334ef628"
format_stamp: "Formatted at 2026-09-29 22:54:10 on dist-test-slave-98mk"
server_key: "b4df135b603a7325d9c096c45b11c785"
server_key_iv: "efdc9ff3c16a1388d3193c3757832522"
server_key_version: "encryptionkey@0"
I20260929 22:54:10.716223 1825 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260929 22:54:10.717044 1840 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 22:54:10.717204 1825 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260929 22:54:10.717288 1825 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790722213368738-14927-0/minicluster-data/ts-1/data,/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790722213368738-14927-0/minicluster-data/ts-1/wal
uuid: "8f1ac88655df49388d45f319334ef628"
format_stamp: "Formatted at 2026-09-29 22:54:10 on dist-test-slave-98mk"
server_key: "b4df135b603a7325d9c096c45b11c785"
server_key_iv: "efdc9ff3c16a1388d3193c3757832522"
server_key_version: "encryptionkey@0"
I20260929 22:54:10.717378 1825 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790722213368738-14927-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790722213368738-14927-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790722213368738-14927-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 22:54:10.733343 1825 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 22:54:10.735183 1825 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 22:54:10.735306 1825 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 22:54:10.735600 1825 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 22:54:10.735904 1825 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 22:54:10.735937 1825 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 22:54:10.735961 1825 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 22:54:10.735984 1825 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 22:54:10.741609 1825 rpc_server.cc:307] RPC server started. Bound to: 127.14.147.194:46789
I20260929 22:54:10.741662 1953 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.147.194:46789 every 8 connection(s)
I20260929 22:54:10.741964 1825 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790722213368738-14927-0/minicluster-data/ts-1/data/info.pb
I20260929 22:54:10.745362 14927 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhWOCLk/build/release/bin/kudu as pid 1825
I20260929 22:54:10.745483 14927 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790722213368738-14927-0/minicluster-data/ts-1/wal/instance
I20260929 22:54:10.745673 14927 external_mini_cluster.cc:1428] Setting key 9ef539714a10590ff3eabcee713bedaf
Sep 29 22:54:10 dist-test-slave-98mk krb5kdc[1583](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790722450, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.147.194@KRBTEST.COM for kudu/127.14.147.254@KRBTEST.COM
I20260929 22:54:10.751684 1668 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:54:10.742645 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.147.194:41169 (local address 127.14.147.254:33425)
0929 22:54:10.742714 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0929 22:54:10.742716 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 22:54:10.742874 (+ 158us) server_negotiation.cc:407] Connection header received
0929 22:54:10.743342 (+ 468us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 22:54:10.743343 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 22:54:10.743352 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 22:54:10.743449 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0929 22:54:10.744394 (+ 945us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:54:10.744891 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:54:10.746166 (+ 1275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:54:10.746407 (+ 241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:54:10.746467 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 22:54:10.748663 (+ 2196us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 22:54:10.748665 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 22:54:10.748665 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 22:54:10.748672 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0929 22:54:10.750204 (+ 1532us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 22:54:10.750619 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 22:54:10.750620 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 22:54:10.750621 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 22:54:10.750680 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 22:54:10.750967 (+ 287us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 22:54:10.750968 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 22:54:10.750968 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 22:54:10.751113 (+ 145us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 22:54:10.751183 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0929 22:54:10.751500 (+ 317us) server_negotiation.cc:299] Negotiation successful
0929 22:54:10.751587 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260929 22:54:10.751830 1956 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:54:10.742600 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.147.254:33425 (local address 127.14.147.194:41169)
0929 22:54:10.742757 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0929 22:54:10.742798 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0929 22:54:10.743270 (+ 472us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 22:54:10.743518 (+ 248us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 22:54:10.743522 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 22:54:10.744082 (+ 560us) client_negotiation.cc:189] Negotiated authn=SASL
0929 22:54:10.744282 (+ 200us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 22:54:10.744287 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 22:54:10.744967 (+ 680us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 22:54:10.744968 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 22:54:10.746091 (+ 1123us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 22:54:10.746095 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 22:54:10.746486 (+ 391us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 22:54:10.746487 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 22:54:10.746622 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 22:54:10.747126 (+ 504us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 22:54:10.747134 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0929 22:54:10.748570 (+ 1436us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 22:54:10.750302 (+ 1732us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 22:54:10.750309 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 22:54:10.750311 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 22:54:10.750544 (+ 233us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 22:54:10.750774 (+ 230us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 22:54:10.750775 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 22:54:10.750776 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 22:54:10.750901 (+ 125us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 22:54:10.751185 (+ 284us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 22:54:10.751190 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 22:54:10.751326 (+ 136us) client_negotiation.cc:769] Sending connection context
0929 22:54:10.751484 (+ 158us) client_negotiation.cc:240] Negotiation successful
0929 22:54:10.751635 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":84,"threads_started":1}
I20260929 22:54:10.752281 1954 heartbeater.cc:347] Connected to a master server at 127.14.147.254:33425
I20260929 22:54:10.752377 1954 heartbeater.cc:464] Registering TS with master...
I20260929 22:54:10.752550 1954 heartbeater.cc:511] Master 127.14.147.254:33425 requested a full tablet report, sending...
I20260929 22:54:10.752897 1628 ts_manager.cc:194] Registered new tserver with Master: 8f1ac88655df49388d45f319334ef628 (127.14.147.194:46789)
I20260929 22:54:10.753237 1628 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.147.194@KRBTEST.COM'} at 127.14.147.194:41169
WARNING: no policy specified for kudu/127.14.147.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.147.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.147.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790722213368738-14927-0/krb5kdc/kudu_127.14.147.195.keytab.
Entry for principal kudu/127.14.147.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790722213368738-14927-0/krb5kdc/kudu_127.14.147.195.keytab.
I20260929 22:54:10.769954 14927 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhWOCLk/build/release/bin/kudu
/tmp/dist-test-taskhWOCLk/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790722213368738-14927-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790722213368738-14927-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790722213368738-14927-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790722213368738-14927-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.147.195:0
--local_ip_for_outbound_sockets=127.14.147.195
--webserver_interface=127.14.147.195
--webserver_port=0
--tserver_master_addrs=127.14.147.254:33425
--keytab_file=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790722213368738-14927-0/krb5kdc/kudu_127.14.147.195.keytab
--principal=kudu/127.14.147.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.147.212:46383
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790722213368738-14927-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790722213368738-14927-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790722213368738-14927-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 22:54:10.851866 1959 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 22:54:10.852102 1959 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 22:54:10.852137 1959 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 22:54:10.853413 1959 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 22:54:10.853461 1959 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 22:54:10.853500 1959 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.147.195
I20260929 22:54:10.855115 1959 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.147.212:46383
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790722213368738-14927-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790722213368738-14927-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790722213368738-14927-0/krb5kdc/kudu_127.14.147.195.keytab
--principal=kudu/127.14.147.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.147.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790722213368738-14927-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.14.147.195
--webserver_port=0
--tserver_master_addrs=127.14.147.254:33425
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1959
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.147.195
--log_dir=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790722213368738-14927-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 21:58:54 UTC on ccycloud.quasar-xguoti.root.comops.site
build id 4681390
I20260929 22:54:10.855381 1959 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 22:54:10.855628 1959 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 22:54:10.860666 1964 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 22:54:10.860658 1967 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 22:54:10.860698 1959 server_base.cc:1034] running on GCE node
W20260929 22:54:10.860661 1965 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 22:54:10.861129 1959 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 22:54:10.861363 1959 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 22:54:10.862519 1959 hybrid_clock.cc:648] HybridClock initialized: now 1790722450862503 us; error 30 us; skew 500 ppm
Sep 29 22:54:10 dist-test-slave-98mk krb5kdc[1583](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790722450, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.147.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 22:54:10.865172 1959 init.cc:377] Logged in from keytab as kudu/127.14.147.195@KRBTEST.COM (short username kudu)
I20260929 22:54:10.865597 1959 webserver.cc:460] Webserver started at http://127.14.147.195:42515/ using document root <none> and password file <none>
I20260929 22:54:10.865803 1959 fs_manager.cc:359] Metadata directory not provided
I20260929 22:54:10.865851 1959 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 22:54:10.865973 1959 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 22:54:10.866986 1959 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790722213368738-14927-0/minicluster-data/ts-2/data/instance:
uuid: "6d00138fa136469da68aa61836ddc783"
format_stamp: "Formatted at 2026-09-29 22:54:10 on dist-test-slave-98mk"
server_key: "64ae2709647564b84c13dcaa36a52440"
server_key_iv: "2ad220da93609b328ed1c09b950f6058"
server_key_version: "encryptionkey@0"
I20260929 22:54:10.867306 1959 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790722213368738-14927-0/minicluster-data/ts-2/wal/instance:
uuid: "6d00138fa136469da68aa61836ddc783"
format_stamp: "Formatted at 2026-09-29 22:54:10 on dist-test-slave-98mk"
server_key: "64ae2709647564b84c13dcaa36a52440"
server_key_iv: "2ad220da93609b328ed1c09b950f6058"
server_key_version: "encryptionkey@0"
I20260929 22:54:10.868589 1959 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260929 22:54:10.869331 1974 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 22:54:10.869499 1959 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260929 22:54:10.869576 1959 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790722213368738-14927-0/minicluster-data/ts-2/data,/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790722213368738-14927-0/minicluster-data/ts-2/wal
uuid: "6d00138fa136469da68aa61836ddc783"
format_stamp: "Formatted at 2026-09-29 22:54:10 on dist-test-slave-98mk"
server_key: "64ae2709647564b84c13dcaa36a52440"
server_key_iv: "2ad220da93609b328ed1c09b950f6058"
server_key_version: "encryptionkey@0"
I20260929 22:54:10.869638 1959 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790722213368738-14927-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790722213368738-14927-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790722213368738-14927-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 22:54:10.902129 1959 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 22:54:10.904295 1959 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 22:54:10.904452 1959 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 22:54:10.904680 1959 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 22:54:10.905020 1959 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 22:54:10.905066 1959 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 22:54:10.905097 1959 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 22:54:10.905121 1959 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 22:54:10.910558 1959 rpc_server.cc:307] RPC server started. Bound to: 127.14.147.195:42045
I20260929 22:54:10.910595 2088 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.147.195:42045 every 8 connection(s)
I20260929 22:54:10.910920 1959 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790722213368738-14927-0/minicluster-data/ts-2/data/info.pb
I20260929 22:54:10.915253 14927 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhWOCLk/build/release/bin/kudu as pid 1959
I20260929 22:54:10.915340 14927 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790722213368738-14927-0/minicluster-data/ts-2/wal/instance
I20260929 22:54:10.915519 14927 external_mini_cluster.cc:1428] Setting key 4e840d234e5f4e926639f6801c8f0e6a
Sep 29 22:54:10 dist-test-slave-98mk krb5kdc[1583](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790722450, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.147.195@KRBTEST.COM for kudu/127.14.147.254@KRBTEST.COM
I20260929 22:54:10.919512 1668 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:54:10.911626 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.147.195:49847 (local address 127.14.147.254:33425)
0929 22:54:10.911676 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0929 22:54:10.911678 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 22:54:10.911909 (+ 231us) server_negotiation.cc:407] Connection header received
0929 22:54:10.912382 (+ 473us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 22:54:10.912383 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 22:54:10.912395 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 22:54:10.912454 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0929 22:54:10.913389 (+ 935us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:54:10.913887 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:54:10.914840 (+ 953us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:54:10.915042 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:54:10.915096 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 22:54:10.917225 (+ 2129us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 22:54:10.917227 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 22:54:10.917227 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 22:54:10.917234 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0929 22:54:10.918258 (+ 1024us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 22:54:10.918611 (+ 353us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 22:54:10.918613 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 22:54:10.918613 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 22:54:10.918665 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 22:54:10.918870 (+ 205us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 22:54:10.918870 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 22:54:10.918871 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 22:54:10.919008 (+ 137us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 22:54:10.919064 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0929 22:54:10.919319 (+ 255us) server_negotiation.cc:299] Negotiation successful
0929 22:54:10.919398 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260929 22:54:10.919700 2091 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:54:10.911614 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.147.254:33425 (local address 127.14.147.195:49847)
0929 22:54:10.911771 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0929 22:54:10.911806 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0929 22:54:10.912292 (+ 486us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 22:54:10.912524 (+ 232us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 22:54:10.912528 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 22:54:10.913097 (+ 569us) client_negotiation.cc:189] Negotiated authn=SASL
0929 22:54:10.913290 (+ 193us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 22:54:10.913295 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 22:54:10.913966 (+ 671us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 22:54:10.913967 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 22:54:10.914763 (+ 796us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 22:54:10.914765 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 22:54:10.915110 (+ 345us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 22:54:10.915111 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 22:54:10.915227 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 22:54:10.915791 (+ 564us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 22:54:10.915800 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0929 22:54:10.917141 (+ 1341us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 22:54:10.918328 (+ 1187us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 22:54:10.918334 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 22:54:10.918336 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 22:54:10.918548 (+ 212us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 22:54:10.918715 (+ 167us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 22:54:10.918716 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 22:54:10.918716 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 22:54:10.918817 (+ 101us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 22:54:10.919062 (+ 245us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 22:54:10.919069 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 22:54:10.919203 (+ 134us) client_negotiation.cc:769] Sending connection context
0929 22:54:10.919290 (+ 87us) client_negotiation.cc:240] Negotiation successful
0929 22:54:10.919491 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":91,"threads_started":1}
I20260929 22:54:10.920125 2089 heartbeater.cc:347] Connected to a master server at 127.14.147.254:33425
I20260929 22:54:10.920202 2089 heartbeater.cc:464] Registering TS with master...
I20260929 22:54:10.920344 2089 heartbeater.cc:511] Master 127.14.147.254:33425 requested a full tablet report, sending...
I20260929 22:54:10.920666 1628 ts_manager.cc:194] Registered new tserver with Master: 6d00138fa136469da68aa61836ddc783 (127.14.147.195:42045)
I20260929 22:54:10.921025 1628 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.147.195@KRBTEST.COM'} at 127.14.147.195:49847
I20260929 22:54:10.927363 14927 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 22:54:10.934166 1668 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:54:10.928640 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46254 (local address 127.14.147.254:33425)
0929 22:54:10.928699 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0929 22:54:10.928701 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 22:54:10.928776 (+ 75us) server_negotiation.cc:407] Connection header received
0929 22:54:10.928848 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 22:54:10.928849 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 22:54:10.928859 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 22:54:10.928900 (+ 41us) server_negotiation.cc:226] Negotiated authn=SASL
0929 22:54:10.929417 (+ 517us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:54:10.929903 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:54:10.930589 (+ 686us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:54:10.930790 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:54:10.930847 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 22:54:10.932038 (+ 1191us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 22:54:10.932040 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 22:54:10.932040 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 22:54:10.932048 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0929 22:54:10.933100 (+ 1052us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 22:54:10.933405 (+ 305us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 22:54:10.933406 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 22:54:10.933406 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 22:54:10.933457 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 22:54:10.933613 (+ 156us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 22:54:10.933614 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 22:54:10.933614 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 22:54:10.933776 (+ 162us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 22:54:10.933865 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0929 22:54:10.933991 (+ 126us) server_negotiation.cc:299] Negotiation successful
0929 22:54:10.934072 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260929 22:54:10.935536 1628 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46254:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 22:54:10.937194 1628 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 22:54:10.944006 2103 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:54:10.939821 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.147.195:42045 (local address 127.0.0.1:33860)
0929 22:54:10.940285 (+ 464us) negotiation.cc:107] Waiting for socket to connect
0929 22:54:10.940294 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0929 22:54:10.940351 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 22:54:10.940509 (+ 158us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 22:54:10.940509 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 22:54:10.940511 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 22:54:10.940692 (+ 181us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 22:54:10.940695 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 22:54:10.941688 (+ 993us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 22:54:10.941689 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 22:54:10.942954 (+ 1265us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 22:54:10.942957 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 22:54:10.943696 (+ 739us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 22:54:10.943697 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 22:54:10.943779 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 22:54:10.943793 (+ 14us) client_negotiation.cc:769] Sending connection context
0929 22:54:10.943859 (+ 66us) client_negotiation.cc:240] Negotiation successful
0929 22:54:10.943913 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":442,"mutex_wait_us":2,"thread_start_us":83,"threads_started":1}
I20260929 22:54:10.944015 2102 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:54:10.939931 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33860 (local address 127.14.147.195:42045)
0929 22:54:10.940109 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0929 22:54:10.940114 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 22:54:10.940346 (+ 232us) server_negotiation.cc:407] Connection header received
0929 22:54:10.940420 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 22:54:10.940423 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 22:54:10.940439 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 22:54:10.940507 (+ 68us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 22:54:10.940758 (+ 251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:54:10.941600 (+ 842us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:54:10.943043 (+ 1443us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:54:10.943623 (+ 580us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:54:10.943692 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 22:54:10.943755 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0929 22:54:10.943889 (+ 134us) server_negotiation.cc:299] Negotiation successful
0929 22:54:10.943932 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":140,"thread_start_us":92,"threads_started":1}
I20260929 22:54:10.944276 2100 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:54:10.940000 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.147.193:38915 (local address 127.0.0.1:47942)
0929 22:54:10.940557 (+ 557us) negotiation.cc:107] Waiting for socket to connect
0929 22:54:10.940567 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0929 22:54:10.940617 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 22:54:10.941032 (+ 415us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 22:54:10.941033 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 22:54:10.941035 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 22:54:10.941078 (+ 43us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 22:54:10.941080 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 22:54:10.942206 (+ 1126us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 22:54:10.942207 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 22:54:10.943077 (+ 870us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 22:54:10.943079 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 22:54:10.944004 (+ 925us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 22:54:10.944005 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 22:54:10.944092 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 22:54:10.944096 (+ 4us) client_negotiation.cc:769] Sending connection context
0929 22:54:10.944146 (+ 50us) client_negotiation.cc:240] Negotiation successful
0929 22:54:10.944192 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":545,"mutex_wait_us":1,"thread_start_us":53,"threads_started":1}
I20260929 22:54:10.944543 2104 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:54:10.940122 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47942 (local address 127.14.147.193:38915)
0929 22:54:10.940270 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0929 22:54:10.940274 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 22:54:10.940817 (+ 543us) server_negotiation.cc:407] Connection header received
0929 22:54:10.940865 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 22:54:10.940868 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 22:54:10.940882 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 22:54:10.940954 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 22:54:10.941155 (+ 201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:54:10.942121 (+ 966us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:54:10.943150 (+ 1029us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:54:10.943850 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:54:10.943919 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 22:54:10.943968 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0929 22:54:10.944413 (+ 445us) server_negotiation.cc:299] Negotiation successful
0929 22:54:10.944465 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":119,"thread_start_us":79,"threads_started":1}
I20260929 22:54:10.944736 2105 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:54:10.940137 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59184 (local address 127.14.147.194:46789)
0929 22:54:10.940324 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0929 22:54:10.940329 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 22:54:10.940339 (+ 10us) server_negotiation.cc:407] Connection header received
0929 22:54:10.940377 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 22:54:10.940379 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 22:54:10.940394 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 22:54:10.940464 (+ 70us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 22:54:10.940630 (+ 166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:54:10.941851 (+ 1221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:54:10.943075 (+ 1224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:54:10.943846 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:54:10.943919 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 22:54:10.943968 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0929 22:54:10.944603 (+ 635us) server_negotiation.cc:299] Negotiation successful
0929 22:54:10.944656 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":158,"thread_start_us":75,"threads_started":1}
I20260929 22:54:10.944778 2023 tablet_service.cc:1467] Processing CreateTablet for tablet 853700dd54ec4eec85bd122f39f3318e (DEFAULT_TABLE table=test-table [id=af49a2ff0f64492ba61ad569886c0a80]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 22:54:10.944993 2101 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:54:10.939784 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.147.194:46789 (local address 127.0.0.1:59184)
0929 22:54:10.940132 (+ 348us) negotiation.cc:107] Waiting for socket to connect
0929 22:54:10.940176 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0929 22:54:10.940259 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 22:54:10.940477 (+ 218us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 22:54:10.940480 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 22:54:10.940483 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 22:54:10.940558 (+ 75us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 22:54:10.940561 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 22:54:10.941951 (+ 1390us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 22:54:10.941952 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 22:54:10.942998 (+ 1046us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 22:54:10.943000 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 22:54:10.944006 (+ 1006us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 22:54:10.944007 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 22:54:10.944520 (+ 513us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 22:54:10.944526 (+ 6us) client_negotiation.cc:769] Sending connection context
0929 22:54:10.944716 (+ 190us) client_negotiation.cc:240] Negotiation successful
0929 22:54:10.944888 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":310,"mutex_wait_us":429,"thread_start_us":68,"threads_started":1}
I20260929 22:54:10.945168 2023 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 853700dd54ec4eec85bd122f39f3318e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 22:54:10.945147 1753 tablet_service.cc:1467] Processing CreateTablet for tablet 853700dd54ec4eec85bd122f39f3318e (DEFAULT_TABLE table=test-table [id=af49a2ff0f64492ba61ad569886c0a80]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 22:54:10.945492 1753 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 853700dd54ec4eec85bd122f39f3318e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 22:54:10.945806 1888 tablet_service.cc:1467] Processing CreateTablet for tablet 853700dd54ec4eec85bd122f39f3318e (DEFAULT_TABLE table=test-table [id=af49a2ff0f64492ba61ad569886c0a80]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 22:54:10.946110 1888 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 853700dd54ec4eec85bd122f39f3318e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 22:54:10.948037 2106 tablet_bootstrap.cc:492] T 853700dd54ec4eec85bd122f39f3318e P b9ae095e7af044e981b5f50ce77481a5: Bootstrap starting.
I20260929 22:54:10.948482 2107 tablet_bootstrap.cc:492] T 853700dd54ec4eec85bd122f39f3318e P 6d00138fa136469da68aa61836ddc783: Bootstrap starting.
I20260929 22:54:10.948804 2106 tablet_bootstrap.cc:654] T 853700dd54ec4eec85bd122f39f3318e P b9ae095e7af044e981b5f50ce77481a5: Neither blocks nor log segments found. Creating new log.
I20260929 22:54:10.949091 2107 tablet_bootstrap.cc:654] T 853700dd54ec4eec85bd122f39f3318e P 6d00138fa136469da68aa61836ddc783: Neither blocks nor log segments found. Creating new log.
I20260929 22:54:10.949260 2108 tablet_bootstrap.cc:492] T 853700dd54ec4eec85bd122f39f3318e P 8f1ac88655df49388d45f319334ef628: Bootstrap starting.
I20260929 22:54:10.949672 2106 log.cc:824] T 853700dd54ec4eec85bd122f39f3318e P b9ae095e7af044e981b5f50ce77481a5: Log is configured to *not* fsync() on all Append() calls
I20260929 22:54:10.949672 2107 log.cc:824] T 853700dd54ec4eec85bd122f39f3318e P 6d00138fa136469da68aa61836ddc783: Log is configured to *not* fsync() on all Append() calls
I20260929 22:54:10.949848 2108 tablet_bootstrap.cc:654] T 853700dd54ec4eec85bd122f39f3318e P 8f1ac88655df49388d45f319334ef628: Neither blocks nor log segments found. Creating new log.
I20260929 22:54:10.950268 2108 log.cc:824] T 853700dd54ec4eec85bd122f39f3318e P 8f1ac88655df49388d45f319334ef628: Log is configured to *not* fsync() on all Append() calls
I20260929 22:54:10.950433 2106 tablet_bootstrap.cc:492] T 853700dd54ec4eec85bd122f39f3318e P b9ae095e7af044e981b5f50ce77481a5: No bootstrap required, opened a new log
I20260929 22:54:10.950496 2106 ts_tablet_manager.cc:1397] T 853700dd54ec4eec85bd122f39f3318e P b9ae095e7af044e981b5f50ce77481a5: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260929 22:54:10.950619 2107 tablet_bootstrap.cc:492] T 853700dd54ec4eec85bd122f39f3318e P 6d00138fa136469da68aa61836ddc783: No bootstrap required, opened a new log
I20260929 22:54:10.950676 2107 ts_tablet_manager.cc:1397] T 853700dd54ec4eec85bd122f39f3318e P 6d00138fa136469da68aa61836ddc783: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260929 22:54:10.951104 2108 tablet_bootstrap.cc:492] T 853700dd54ec4eec85bd122f39f3318e P 8f1ac88655df49388d45f319334ef628: No bootstrap required, opened a new log
I20260929 22:54:10.951171 2108 ts_tablet_manager.cc:1397] T 853700dd54ec4eec85bd122f39f3318e P 8f1ac88655df49388d45f319334ef628: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260929 22:54:10.952111 2106 raft_consensus.cc:348] T 853700dd54ec4eec85bd122f39f3318e P b9ae095e7af044e981b5f50ce77481a5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f1ac88655df49388d45f319334ef628" member_type: VOTER last_known_addr { host: "127.14.147.194" port: 46789 } } peers { permanent_uuid: "6d00138fa136469da68aa61836ddc783" member_type: VOTER last_known_addr { host: "127.14.147.195" port: 42045 } } peers { permanent_uuid: "b9ae095e7af044e981b5f50ce77481a5" member_type: VOTER last_known_addr { host: "127.14.147.193" port: 38915 } }
I20260929 22:54:10.952270 2106 raft_consensus.cc:374] T 853700dd54ec4eec85bd122f39f3318e P b9ae095e7af044e981b5f50ce77481a5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 22:54:10.952303 2106 raft_consensus.cc:729] T 853700dd54ec4eec85bd122f39f3318e P b9ae095e7af044e981b5f50ce77481a5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b9ae095e7af044e981b5f50ce77481a5, State: Initialized, Role: FOLLOWER
I20260929 22:54:10.952418 2106 consensus_queue.cc:260] T 853700dd54ec4eec85bd122f39f3318e P b9ae095e7af044e981b5f50ce77481a5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f1ac88655df49388d45f319334ef628" member_type: VOTER last_known_addr { host: "127.14.147.194" port: 46789 } } peers { permanent_uuid: "6d00138fa136469da68aa61836ddc783" member_type: VOTER last_known_addr { host: "127.14.147.195" port: 42045 } } peers { permanent_uuid: "b9ae095e7af044e981b5f50ce77481a5" member_type: VOTER last_known_addr { host: "127.14.147.193" port: 38915 } }
I20260929 22:54:10.952688 2106 ts_tablet_manager.cc:1428] T 853700dd54ec4eec85bd122f39f3318e P b9ae095e7af044e981b5f50ce77481a5: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260929 22:54:10.952685 1820 heartbeater.cc:503] Master 127.14.147.254:33425 was elected leader, sending a full tablet report...
I20260929 22:54:10.952641 2107 raft_consensus.cc:348] T 853700dd54ec4eec85bd122f39f3318e P 6d00138fa136469da68aa61836ddc783 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f1ac88655df49388d45f319334ef628" member_type: VOTER last_known_addr { host: "127.14.147.194" port: 46789 } } peers { permanent_uuid: "6d00138fa136469da68aa61836ddc783" member_type: VOTER last_known_addr { host: "127.14.147.195" port: 42045 } } peers { permanent_uuid: "b9ae095e7af044e981b5f50ce77481a5" member_type: VOTER last_known_addr { host: "127.14.147.193" port: 38915 } }
I20260929 22:54:10.952996 2107 raft_consensus.cc:374] T 853700dd54ec4eec85bd122f39f3318e P 6d00138fa136469da68aa61836ddc783 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 22:54:10.953037 2107 raft_consensus.cc:729] T 853700dd54ec4eec85bd122f39f3318e P 6d00138fa136469da68aa61836ddc783 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6d00138fa136469da68aa61836ddc783, State: Initialized, Role: FOLLOWER
I20260929 22:54:10.953148 2107 consensus_queue.cc:260] T 853700dd54ec4eec85bd122f39f3318e P 6d00138fa136469da68aa61836ddc783 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f1ac88655df49388d45f319334ef628" member_type: VOTER last_known_addr { host: "127.14.147.194" port: 46789 } } peers { permanent_uuid: "6d00138fa136469da68aa61836ddc783" member_type: VOTER last_known_addr { host: "127.14.147.195" port: 42045 } } peers { permanent_uuid: "b9ae095e7af044e981b5f50ce77481a5" member_type: VOTER last_known_addr { host: "127.14.147.193" port: 38915 } }
I20260929 22:54:10.953150 2108 raft_consensus.cc:348] T 853700dd54ec4eec85bd122f39f3318e P 8f1ac88655df49388d45f319334ef628 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f1ac88655df49388d45f319334ef628" member_type: VOTER last_known_addr { host: "127.14.147.194" port: 46789 } } peers { permanent_uuid: "6d00138fa136469da68aa61836ddc783" member_type: VOTER last_known_addr { host: "127.14.147.195" port: 42045 } } peers { permanent_uuid: "b9ae095e7af044e981b5f50ce77481a5" member_type: VOTER last_known_addr { host: "127.14.147.193" port: 38915 } }
I20260929 22:54:10.953325 2108 raft_consensus.cc:374] T 853700dd54ec4eec85bd122f39f3318e P 8f1ac88655df49388d45f319334ef628 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 22:54:10.953369 2108 raft_consensus.cc:729] T 853700dd54ec4eec85bd122f39f3318e P 8f1ac88655df49388d45f319334ef628 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8f1ac88655df49388d45f319334ef628, State: Initialized, Role: FOLLOWER
I20260929 22:54:10.953429 2107 ts_tablet_manager.cc:1428] T 853700dd54ec4eec85bd122f39f3318e P 6d00138fa136469da68aa61836ddc783: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260929 22:54:10.953533 2089 heartbeater.cc:503] Master 127.14.147.254:33425 was elected leader, sending a full tablet report...
I20260929 22:54:10.953738 2108 consensus_queue.cc:260] T 853700dd54ec4eec85bd122f39f3318e P 8f1ac88655df49388d45f319334ef628 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f1ac88655df49388d45f319334ef628" member_type: VOTER last_known_addr { host: "127.14.147.194" port: 46789 } } peers { permanent_uuid: "6d00138fa136469da68aa61836ddc783" member_type: VOTER last_known_addr { host: "127.14.147.195" port: 42045 } } peers { permanent_uuid: "b9ae095e7af044e981b5f50ce77481a5" member_type: VOTER last_known_addr { host: "127.14.147.193" port: 38915 } }
I20260929 22:54:10.954006 2108 ts_tablet_manager.cc:1428] T 853700dd54ec4eec85bd122f39f3318e P 8f1ac88655df49388d45f319334ef628: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260929 22:54:10.954092 1954 heartbeater.cc:503] Master 127.14.147.254:33425 was elected leader, sending a full tablet report...
W20260929 22:54:10.992525 1955 tablet.cc:2369] T 853700dd54ec4eec85bd122f39f3318e P 8f1ac88655df49388d45f319334ef628: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260929 22:54:11.089098 1821 tablet.cc:2369] T 853700dd54ec4eec85bd122f39f3318e P b9ae095e7af044e981b5f50ce77481a5: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 22:54:11.146353 2112 raft_consensus.cc:482] T 853700dd54ec4eec85bd122f39f3318e P b9ae095e7af044e981b5f50ce77481a5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 22:54:11.146493 2112 raft_consensus.cc:504] T 853700dd54ec4eec85bd122f39f3318e P b9ae095e7af044e981b5f50ce77481a5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f1ac88655df49388d45f319334ef628" member_type: VOTER last_known_addr { host: "127.14.147.194" port: 46789 } } peers { permanent_uuid: "6d00138fa136469da68aa61836ddc783" member_type: VOTER last_known_addr { host: "127.14.147.195" port: 42045 } } peers { permanent_uuid: "b9ae095e7af044e981b5f50ce77481a5" member_type: VOTER last_known_addr { host: "127.14.147.193" port: 38915 } }
I20260929 22:54:11.146811 2112 leader_election.cc:288] T 853700dd54ec4eec85bd122f39f3318e P b9ae095e7af044e981b5f50ce77481a5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8f1ac88655df49388d45f319334ef628 (127.14.147.194:46789), 6d00138fa136469da68aa61836ddc783 (127.14.147.195:42045)
I20260929 22:54:11.150935 2102 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:54:11.147098 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.147.193:34551 (local address 127.14.147.195:42045)
0929 22:54:11.147146 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0929 22:54:11.147149 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 22:54:11.147195 (+ 46us) server_negotiation.cc:407] Connection header received
0929 22:54:11.147264 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 22:54:11.147265 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 22:54:11.147274 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 22:54:11.147348 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 22:54:11.147666 (+ 318us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:54:11.148351 (+ 685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:54:11.149863 (+ 1512us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:54:11.150469 (+ 606us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:54:11.150550 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 22:54:11.150560 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0929 22:54:11.150806 (+ 246us) server_negotiation.cc:299] Negotiation successful
0929 22:54:11.150861 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260929 22:54:11.150935 2105 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:54:11.147136 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.147.193:41379 (local address 127.14.147.194:46789)
0929 22:54:11.147175 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0929 22:54:11.147177 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 22:54:11.147186 (+ 9us) server_negotiation.cc:407] Connection header received
0929 22:54:11.147264 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 22:54:11.147265 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 22:54:11.147276 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 22:54:11.147348 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 22:54:11.147542 (+ 194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:54:11.148290 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:54:11.149865 (+ 1575us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:54:11.150467 (+ 602us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:54:11.150550 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 22:54:11.150560 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0929 22:54:11.150806 (+ 246us) server_negotiation.cc:299] Negotiation successful
0929 22:54:11.150868 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260929 22:54:11.150941 2116 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:54:11.147004 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.147.195:42045 (local address 127.14.147.193:34551)
0929 22:54:11.147138 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0929 22:54:11.147146 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0929 22:54:11.147199 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 22:54:11.147360 (+ 161us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 22:54:11.147361 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 22:54:11.147363 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 22:54:11.147476 (+ 113us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 22:54:11.147477 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 22:54:11.148430 (+ 953us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 22:54:11.148430 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 22:54:11.149746 (+ 1316us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 22:54:11.149749 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 22:54:11.150581 (+ 832us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 22:54:11.150583 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 22:54:11.150680 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 22:54:11.150687 (+ 7us) client_negotiation.cc:769] Sending connection context
0929 22:54:11.150756 (+ 69us) client_negotiation.cc:240] Negotiation successful
0929 22:54:11.150809 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":125,"mutex_wait_us":54,"thread_start_us":62,"threads_started":1}
I20260929 22:54:11.150940 2115 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:54:11.147003 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.147.194:46789 (local address 127.14.147.193:41379)
0929 22:54:11.147132 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0929 22:54:11.147144 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0929 22:54:11.147199 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 22:54:11.147361 (+ 162us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 22:54:11.147362 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 22:54:11.147363 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 22:54:11.147470 (+ 107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 22:54:11.147473 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 22:54:11.148395 (+ 922us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 22:54:11.148397 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 22:54:11.149745 (+ 1348us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 22:54:11.149749 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 22:54:11.150583 (+ 834us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 22:54:11.150583 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 22:54:11.150680 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 22:54:11.150689 (+ 9us) client_negotiation.cc:769] Sending connection context
0929 22:54:11.150756 (+ 67us) client_negotiation.cc:240] Negotiation successful
0929 22:54:11.150813 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":111,"mutex_wait_us":50,"thread_start_us":64,"threads_started":1}
I20260929 22:54:11.151523 2043 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "853700dd54ec4eec85bd122f39f3318e" candidate_uuid: "b9ae095e7af044e981b5f50ce77481a5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6d00138fa136469da68aa61836ddc783" is_pre_election: true
I20260929 22:54:11.151631 1908 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "853700dd54ec4eec85bd122f39f3318e" candidate_uuid: "b9ae095e7af044e981b5f50ce77481a5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f1ac88655df49388d45f319334ef628" is_pre_election: true
I20260929 22:54:11.151767 1908 raft_consensus.cc:2454] T 853700dd54ec4eec85bd122f39f3318e P 8f1ac88655df49388d45f319334ef628 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9ae095e7af044e981b5f50ce77481a5 in term 0.
I20260929 22:54:11.151823 2043 raft_consensus.cc:2454] T 853700dd54ec4eec85bd122f39f3318e P 6d00138fa136469da68aa61836ddc783 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9ae095e7af044e981b5f50ce77481a5 in term 0.
I20260929 22:54:11.152020 1706 leader_election.cc:302] T 853700dd54ec4eec85bd122f39f3318e P b9ae095e7af044e981b5f50ce77481a5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8f1ac88655df49388d45f319334ef628, b9ae095e7af044e981b5f50ce77481a5; no voters:
I20260929 22:54:11.152217 2112 raft_consensus.cc:2781] T 853700dd54ec4eec85bd122f39f3318e P b9ae095e7af044e981b5f50ce77481a5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 22:54:11.152379 2112 raft_consensus.cc:482] T 853700dd54ec4eec85bd122f39f3318e P b9ae095e7af044e981b5f50ce77481a5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 22:54:11.152415 2112 raft_consensus.cc:3037] T 853700dd54ec4eec85bd122f39f3318e P b9ae095e7af044e981b5f50ce77481a5 [term 0 FOLLOWER]: Advancing to term 1
I20260929 22:54:11.153319 2112 raft_consensus.cc:504] T 853700dd54ec4eec85bd122f39f3318e P b9ae095e7af044e981b5f50ce77481a5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f1ac88655df49388d45f319334ef628" member_type: VOTER last_known_addr { host: "127.14.147.194" port: 46789 } } peers { permanent_uuid: "6d00138fa136469da68aa61836ddc783" member_type: VOTER last_known_addr { host: "127.14.147.195" port: 42045 } } peers { permanent_uuid: "b9ae095e7af044e981b5f50ce77481a5" member_type: VOTER last_known_addr { host: "127.14.147.193" port: 38915 } }
I20260929 22:54:11.153479 2112 leader_election.cc:288] T 853700dd54ec4eec85bd122f39f3318e P b9ae095e7af044e981b5f50ce77481a5 [CANDIDATE]: Term 1 election: Requested vote from peers 8f1ac88655df49388d45f319334ef628 (127.14.147.194:46789), 6d00138fa136469da68aa61836ddc783 (127.14.147.195:42045)
I20260929 22:54:11.153702 1908 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "853700dd54ec4eec85bd122f39f3318e" candidate_uuid: "b9ae095e7af044e981b5f50ce77481a5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f1ac88655df49388d45f319334ef628"
I20260929 22:54:11.153797 1908 raft_consensus.cc:3037] T 853700dd54ec4eec85bd122f39f3318e P 8f1ac88655df49388d45f319334ef628 [term 0 FOLLOWER]: Advancing to term 1
I20260929 22:54:11.153816 2043 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "853700dd54ec4eec85bd122f39f3318e" candidate_uuid: "b9ae095e7af044e981b5f50ce77481a5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6d00138fa136469da68aa61836ddc783"
I20260929 22:54:11.153934 2043 raft_consensus.cc:3037] T 853700dd54ec4eec85bd122f39f3318e P 6d00138fa136469da68aa61836ddc783 [term 0 FOLLOWER]: Advancing to term 1
I20260929 22:54:11.154560 1908 raft_consensus.cc:2454] T 853700dd54ec4eec85bd122f39f3318e P 8f1ac88655df49388d45f319334ef628 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b9ae095e7af044e981b5f50ce77481a5 in term 1.
I20260929 22:54:11.154798 1706 leader_election.cc:302] T 853700dd54ec4eec85bd122f39f3318e P b9ae095e7af044e981b5f50ce77481a5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8f1ac88655df49388d45f319334ef628, b9ae095e7af044e981b5f50ce77481a5; no voters:
I20260929 22:54:11.154904 2112 raft_consensus.cc:2781] T 853700dd54ec4eec85bd122f39f3318e P b9ae095e7af044e981b5f50ce77481a5 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 22:54:11.155076 2112 raft_consensus.cc:686] T 853700dd54ec4eec85bd122f39f3318e P b9ae095e7af044e981b5f50ce77481a5 [term 1 LEADER]: Becoming Leader. State: Replica: b9ae095e7af044e981b5f50ce77481a5, State: Running, Role: LEADER
I20260929 22:54:11.155076 2043 raft_consensus.cc:2454] T 853700dd54ec4eec85bd122f39f3318e P 6d00138fa136469da68aa61836ddc783 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b9ae095e7af044e981b5f50ce77481a5 in term 1.
I20260929 22:54:11.155186 2112 consensus_queue.cc:237] T 853700dd54ec4eec85bd122f39f3318e P b9ae095e7af044e981b5f50ce77481a5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f1ac88655df49388d45f319334ef628" member_type: VOTER last_known_addr { host: "127.14.147.194" port: 46789 } } peers { permanent_uuid: "6d00138fa136469da68aa61836ddc783" member_type: VOTER last_known_addr { host: "127.14.147.195" port: 42045 } } peers { permanent_uuid: "b9ae095e7af044e981b5f50ce77481a5" member_type: VOTER last_known_addr { host: "127.14.147.193" port: 38915 } }
I20260929 22:54:11.156262 1628 catalog_manager.cc:5495] T 853700dd54ec4eec85bd122f39f3318e P b9ae095e7af044e981b5f50ce77481a5 reported cstate change: term changed from 0 to 1, leader changed from <none> to b9ae095e7af044e981b5f50ce77481a5 (127.14.147.193). New cstate: current_term: 1 leader_uuid: "b9ae095e7af044e981b5f50ce77481a5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f1ac88655df49388d45f319334ef628" member_type: VOTER last_known_addr { host: "127.14.147.194" port: 46789 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6d00138fa136469da68aa61836ddc783" member_type: VOTER last_known_addr { host: "127.14.147.195" port: 42045 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b9ae095e7af044e981b5f50ce77481a5" member_type: VOTER last_known_addr { host: "127.14.147.193" port: 38915 } health_report { overall_health: HEALTHY } } }
W20260929 22:54:11.161476 2090 tablet.cc:2369] T 853700dd54ec4eec85bd122f39f3318e P 6d00138fa136469da68aa61836ddc783: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 22:54:11.172499 2104 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:54:11.170188 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47944 (local address 127.14.147.193:38915)
0929 22:54:11.170235 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0929 22:54:11.170236 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0929 22:54:11.170244 (+ 8us) server_negotiation.cc:407] Connection header received
0929 22:54:11.170312 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 22:54:11.170313 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 22:54:11.170331 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 22:54:11.170397 (+ 66us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 22:54:11.170503 (+ 106us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:54:11.170969 (+ 466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:54:11.171795 (+ 826us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:54:11.172019 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:54:11.172081 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 22:54:11.172212 (+ 131us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 22:54:11.172324 (+ 112us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 22:54:11.172383 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0929 22:54:11.172413 (+ 30us) server_negotiation.cc:299] Negotiation successful
0929 22:54:11.172434 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260929 22:54:11.174325 2043 raft_consensus.cc:1261] T 853700dd54ec4eec85bd122f39f3318e P 6d00138fa136469da68aa61836ddc783 [term 1 FOLLOWER]: Refusing update from remote peer b9ae095e7af044e981b5f50ce77481a5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 22:54:11.174628 1908 raft_consensus.cc:1261] T 853700dd54ec4eec85bd122f39f3318e P 8f1ac88655df49388d45f319334ef628 [term 1 FOLLOWER]: Refusing update from remote peer b9ae095e7af044e981b5f50ce77481a5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 22:54:11.174664 2117 consensus_queue.cc:1035] T 853700dd54ec4eec85bd122f39f3318e P b9ae095e7af044e981b5f50ce77481a5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d00138fa136469da68aa61836ddc783" member_type: VOTER last_known_addr { host: "127.14.147.195" port: 42045 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 22:54:11.174877 2117 consensus_queue.cc:1035] T 853700dd54ec4eec85bd122f39f3318e P b9ae095e7af044e981b5f50ce77481a5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8f1ac88655df49388d45f319334ef628" member_type: VOTER last_known_addr { host: "127.14.147.194" port: 46789 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 22:54:11.177973 2122 mvcc.cc:205] Tried to move back new op lower bound from 7334799160007495680 to 7334799159932657664. Current Snapshot: MvccSnapshot[applied={T|T < 7334799160007495680}]
I20260929 22:54:11.178005 2121 mvcc.cc:205] Tried to move back new op lower bound from 7334799160007495680 to 7334799159932657664. Current Snapshot: MvccSnapshot[applied={T|T < 7334799160007495680}]
I20260929 22:54:11.180405 1628 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46254:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 22:54:11.180503 1628 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46254:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 22:54:11.181238 1628 catalog_manager.cc:5782] T 00000000000000000000000000000000 P e51dbed0f1914a609f10ad8a571d5cec: Sending DeleteTablet for 3 replicas of tablet 853700dd54ec4eec85bd122f39f3318e
I20260929 22:54:11.181520 1888 tablet_service.cc:1514] Processing DeleteTablet for tablet 853700dd54ec4eec85bd122f39f3318e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 22:54:11 UTC) from {username='kudu', principal='kudu/127.14.147.254@KRBTEST.COM'} at 127.0.0.1:59184
I20260929 22:54:11.181653 2023 tablet_service.cc:1514] Processing DeleteTablet for tablet 853700dd54ec4eec85bd122f39f3318e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 22:54:11 UTC) from {username='kudu', principal='kudu/127.14.147.254@KRBTEST.COM'} at 127.0.0.1:33860
I20260929 22:54:11.181811 2130 tablet_replica.cc:330] T 853700dd54ec4eec85bd122f39f3318e P 8f1ac88655df49388d45f319334ef628: stopping tablet replica
I20260929 22:54:11.181849 14927 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhWOCLk/build/release/bin/kudu with pid 1690
I20260929 22:54:11.181874 1753 tablet_service.cc:1514] Processing DeleteTablet for tablet 853700dd54ec4eec85bd122f39f3318e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 22:54:11 UTC) from {username='kudu', principal='kudu/127.14.147.254@KRBTEST.COM'} at 127.0.0.1:47942
I20260929 22:54:11.181910 2130 raft_consensus.cc:2229] T 853700dd54ec4eec85bd122f39f3318e P 8f1ac88655df49388d45f319334ef628 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 22:54:11.182006 2130 raft_consensus.cc:2258] T 853700dd54ec4eec85bd122f39f3318e P 8f1ac88655df49388d45f319334ef628 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 22:54:11.182427 2131 tablet_replica.cc:330] T 853700dd54ec4eec85bd122f39f3318e P 6d00138fa136469da68aa61836ddc783: stopping tablet replica
I20260929 22:54:11.182523 2130 ts_tablet_manager.cc:1905] T 853700dd54ec4eec85bd122f39f3318e P 8f1ac88655df49388d45f319334ef628: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 22:54:11.182548 2131 raft_consensus.cc:2229] T 853700dd54ec4eec85bd122f39f3318e P 6d00138fa136469da68aa61836ddc783 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 22:54:11.182646 2131 raft_consensus.cc:2258] T 853700dd54ec4eec85bd122f39f3318e P 6d00138fa136469da68aa61836ddc783 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 22:54:11.184125 2130 ts_tablet_manager.cc:1918] T 853700dd54ec4eec85bd122f39f3318e P 8f1ac88655df49388d45f319334ef628: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 22:54:11.184201 2130 log.cc:1196] T 853700dd54ec4eec85bd122f39f3318e P 8f1ac88655df49388d45f319334ef628: Deleting WAL directory at /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790722213368738-14927-0/minicluster-data/ts-1/wal/wals/853700dd54ec4eec85bd122f39f3318e
I20260929 22:54:11.184530 2130 ts_tablet_manager.cc:1939] T 853700dd54ec4eec85bd122f39f3318e P 8f1ac88655df49388d45f319334ef628: Deleting consensus metadata
I20260929 22:54:11.184962 1612 catalog_manager.cc:4841] TS 8f1ac88655df49388d45f319334ef628 (127.14.147.194:46789): tablet 853700dd54ec4eec85bd122f39f3318e (table test-table [id=af49a2ff0f64492ba61ad569886c0a80]) successfully deleted
W20260929 22:54:11.187109 1615 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.14.147.193:38915 (error 108)
I20260929 22:54:11.187619 2131 ts_tablet_manager.cc:1905] T 853700dd54ec4eec85bd122f39f3318e P 6d00138fa136469da68aa61836ddc783: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260929 22:54:11.189014 2100 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 22:54:11.187547 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.147.193:38915 (local address 127.0.0.1:47960)
0929 22:54:11.187602 (+ 55us) negotiation.cc:107] Waiting for socket to connect
0929 22:54:11.187613 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0929 22:54:11.187656 (+ 43us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 22:54:11.188960 (+ 1304us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.147.193:38915: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":31}
I20260929 22:54:11.189028 2131 ts_tablet_manager.cc:1918] T 853700dd54ec4eec85bd122f39f3318e P 6d00138fa136469da68aa61836ddc783: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 22:54:11.189091 2131 log.cc:1196] T 853700dd54ec4eec85bd122f39f3318e P 6d00138fa136469da68aa61836ddc783: Deleting WAL directory at /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790722213368738-14927-0/minicluster-data/ts-2/wal/wals/853700dd54ec4eec85bd122f39f3318e
I20260929 22:54:11.189167 14927 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhWOCLk/build/release/bin/kudu with pid 1825
W20260929 22:54:11.189172 1615 catalog_manager.cc:4584] TS b9ae095e7af044e981b5f50ce77481a5 (127.14.147.193:38915): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 853700dd54ec4eec85bd122f39f3318e: Network error: Client connection negotiation failed: client connection to 127.14.147.193:38915: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260929 22:54:11.189359 2131 ts_tablet_manager.cc:1939] T 853700dd54ec4eec85bd122f39f3318e P 6d00138fa136469da68aa61836ddc783: Deleting consensus metadata
I20260929 22:54:11.190104 1615 catalog_manager.cc:4841] TS 6d00138fa136469da68aa61836ddc783 (127.14.147.195:42045): tablet 853700dd54ec4eec85bd122f39f3318e (table test-table [id=af49a2ff0f64492ba61ad569886c0a80]) successfully deleted
I20260929 22:54:11.194286 14927 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhWOCLk/build/release/bin/kudu with pid 1959
I20260929 22:54:11.198315 14927 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhWOCLk/build/release/bin/kudu with pid 1596
2026-09-29T22:54:11Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/4 (2006 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/5
2026-09-29T22:54:11Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T22:54:11Z Disabled control of system clock
I20260929 22:54:11.223757 14927 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhWOCLk/build/release/bin/kudu
/tmp/dist-test-taskhWOCLk/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790722213368738-14927-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790722213368738-14927-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790722213368738-14927-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790722213368738-14927-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.147.254:43041
--webserver_interface=127.14.147.254
--webserver_port=0
--builtin_ntp_servers=127.14.147.212:40105
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.147.254:43041
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260929 22:54:11.302587 2136 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 22:54:11.302826 2136 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 22:54:11.302850 2136 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 22:54:11.304014 2136 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 22:54:11.304055 2136 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 22:54:11.304067 2136 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 22:54:11.304078 2136 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 22:54:11.304088 2136 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 22:54:11.305423 2136 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.147.212:40105
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790722213368738-14927-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790722213368738-14927-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.14.147.254:43041
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.147.254:43041
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790722213368738-14927-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.14.147.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2136
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790722213368738-14927-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 21:58:54 UTC on ccycloud.quasar-xguoti.root.comops.site
build id 4681390
I20260929 22:54:11.305639 2136 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 22:54:11.305891 2136 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 22:54:11.310443 2142 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 22:54:11.310575 2136 server_base.cc:1034] running on GCE node
W20260929 22:54:11.310423 2141 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 22:54:11.310423 2144 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 22:54:11.310915 2136 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 22:54:11.311179 2136 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 22:54:11.312311 2136 hybrid_clock.cc:648] HybridClock initialized: now 1790722451312301 us; error 28 us; skew 500 ppm
I20260929 22:54:11.313534 2136 webserver.cc:460] Webserver started at http://127.14.147.254:46539/ using document root <none> and password file <none>
I20260929 22:54:11.313778 2136 fs_manager.cc:359] Metadata directory not provided
I20260929 22:54:11.313825 2136 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 22:54:11.313958 2136 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 22:54:11.315021 2136 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790722213368738-14927-0/minicluster-data/master-0/data/instance:
uuid: "c5174aff7fe34574aa07ea9e5dd38d69"
format_stamp: "Formatted at 2026-09-29 22:54:11 on dist-test-slave-98mk"
server_key: "3b7eb3743655423e410762d26fb594cd"
server_key_iv: "ba01401ae91405288481c3d19a7ee633"
server_key_version: "encryptionkey@0"
I20260929 22:54:11.315353 2136 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790722213368738-14927-0/minicluster-data/master-0/wal/instance:
uuid: "c5174aff7fe34574aa07ea9e5dd38d69"
format_stamp: "Formatted at 2026-09-29 22:54:11 on dist-test-slave-98mk"
server_key: "3b7eb3743655423e410762d26fb594cd"
server_key_iv: "ba01401ae91405288481c3d19a7ee633"
server_key_version: "encryptionkey@0"
I20260929 22:54:11.316596 2136 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260929 22:54:11.317337 2150 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 22:54:11.317510 2136 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260929 22:54:11.317591 2136 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790722213368738-14927-0/minicluster-data/master-0/data,/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790722213368738-14927-0/minicluster-data/master-0/wal
uuid: "c5174aff7fe34574aa07ea9e5dd38d69"
format_stamp: "Formatted at 2026-09-29 22:54:11 on dist-test-slave-98mk"
server_key: "3b7eb3743655423e410762d26fb594cd"
server_key_iv: "ba01401ae91405288481c3d19a7ee633"
server_key_version: "encryptionkey@0"
I20260929 22:54:11.317654 2136 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790722213368738-14927-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790722213368738-14927-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790722213368738-14927-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 22:54:11.331099 2136 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 22:54:11.331462 2136 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 22:54:11.331604 2136 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 22:54:11.335261 2136 rpc_server.cc:307] RPC server started. Bound to: 127.14.147.254:43041
I20260929 22:54:11.335297 2202 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.147.254:43041 every 8 connection(s)
I20260929 22:54:11.335656 2136 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790722213368738-14927-0/minicluster-data/master-0/data/info.pb
I20260929 22:54:11.336663 2203 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 22:54:11.337852 14927 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhWOCLk/build/release/bin/kudu as pid 2136
I20260929 22:54:11.337949 14927 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790722213368738-14927-0/minicluster-data/master-0/wal/instance
I20260929 22:54:11.338119 14927 external_mini_cluster.cc:1428] Setting key 1154995e1c7f68146b2d48f8459fbee7
I20260929 22:54:11.339037 2203 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c5174aff7fe34574aa07ea9e5dd38d69: Bootstrap starting.
I20260929 22:54:11.339614 2203 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c5174aff7fe34574aa07ea9e5dd38d69: Neither blocks nor log segments found. Creating new log.
I20260929 22:54:11.339869 2203 log.cc:824] T 00000000000000000000000000000000 P c5174aff7fe34574aa07ea9e5dd38d69: Log is configured to *not* fsync() on all Append() calls
I20260929 22:54:11.340479 2203 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c5174aff7fe34574aa07ea9e5dd38d69: No bootstrap required, opened a new log
I20260929 22:54:11.342278 2203 raft_consensus.cc:348] T 00000000000000000000000000000000 P c5174aff7fe34574aa07ea9e5dd38d69 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c5174aff7fe34574aa07ea9e5dd38d69" member_type: VOTER last_known_addr { host: "127.14.147.254" port: 43041 } }
I20260929 22:54:11.342411 2203 raft_consensus.cc:374] T 00000000000000000000000000000000 P c5174aff7fe34574aa07ea9e5dd38d69 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 22:54:11.342442 2203 raft_consensus.cc:729] T 00000000000000000000000000000000 P c5174aff7fe34574aa07ea9e5dd38d69 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c5174aff7fe34574aa07ea9e5dd38d69, State: Initialized, Role: FOLLOWER
I20260929 22:54:11.342592 2203 consensus_queue.cc:260] T 00000000000000000000000000000000 P c5174aff7fe34574aa07ea9e5dd38d69 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c5174aff7fe34574aa07ea9e5dd38d69" member_type: VOTER last_known_addr { host: "127.14.147.254" port: 43041 } }
I20260929 22:54:11.342696 2203 raft_consensus.cc:388] T 00000000000000000000000000000000 P c5174aff7fe34574aa07ea9e5dd38d69 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 22:54:11.342729 2203 raft_consensus.cc:482] T 00000000000000000000000000000000 P c5174aff7fe34574aa07ea9e5dd38d69 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 22:54:11.342809 2203 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c5174aff7fe34574aa07ea9e5dd38d69 [term 0 FOLLOWER]: Advancing to term 1
I20260929 22:54:11.343341 2206 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:54:11.338617 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59268 (local address 127.14.147.254:43041)
0929 22:54:11.338802 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0929 22:54:11.338811 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 22:54:11.338845 (+ 34us) server_negotiation.cc:407] Connection header received
0929 22:54:11.339291 (+ 446us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 22:54:11.339297 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 22:54:11.339357 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 22:54:11.339545 (+ 188us) server_negotiation.cc:226] Negotiated authn=SASL
0929 22:54:11.339772 (+ 227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:54:11.340666 (+ 894us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:54:11.341405 (+ 739us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:54:11.341828 (+ 423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:54:11.341932 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 22:54:11.342560 (+ 628us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 22:54:11.342564 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 22:54:11.342564 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 22:54:11.342779 (+ 215us) server_negotiation.cc:922] calling sasl_server_start()
0929 22:54:11.342824 (+ 45us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 22:54:11.342840 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 22:54:11.342912 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0929 22:54:11.343019 (+ 107us) server_negotiation.cc:299] Negotiation successful
0929 22:54:11.343094 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":99,"thread_start_us":57,"threads_started":1}
I20260929 22:54:11.343758 2203 raft_consensus.cc:504] T 00000000000000000000000000000000 P c5174aff7fe34574aa07ea9e5dd38d69 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c5174aff7fe34574aa07ea9e5dd38d69" member_type: VOTER last_known_addr { host: "127.14.147.254" port: 43041 } }
I20260929 22:54:11.343918 2203 leader_election.cc:302] T 00000000000000000000000000000000 P c5174aff7fe34574aa07ea9e5dd38d69 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c5174aff7fe34574aa07ea9e5dd38d69; no voters:
I20260929 22:54:11.344075 2203 leader_election.cc:288] T 00000000000000000000000000000000 P c5174aff7fe34574aa07ea9e5dd38d69 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 22:54:11.344103 2208 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c5174aff7fe34574aa07ea9e5dd38d69 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 22:54:11.344241 2208 raft_consensus.cc:686] T 00000000000000000000000000000000 P c5174aff7fe34574aa07ea9e5dd38d69 [term 1 LEADER]: Becoming Leader. State: Replica: c5174aff7fe34574aa07ea9e5dd38d69, State: Running, Role: LEADER
I20260929 22:54:11.344321 2203 sys_catalog.cc:564] T 00000000000000000000000000000000 P c5174aff7fe34574aa07ea9e5dd38d69 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 22:54:11.344326 2208 consensus_queue.cc:237] T 00000000000000000000000000000000 P c5174aff7fe34574aa07ea9e5dd38d69 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c5174aff7fe34574aa07ea9e5dd38d69" member_type: VOTER last_known_addr { host: "127.14.147.254" port: 43041 } }
I20260929 22:54:11.344704 2209 sys_catalog.cc:455] T 00000000000000000000000000000000 P c5174aff7fe34574aa07ea9e5dd38d69 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c5174aff7fe34574aa07ea9e5dd38d69" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c5174aff7fe34574aa07ea9e5dd38d69" member_type: VOTER last_known_addr { host: "127.14.147.254" port: 43041 } } }
I20260929 22:54:11.344799 2209 sys_catalog.cc:458] T 00000000000000000000000000000000 P c5174aff7fe34574aa07ea9e5dd38d69 [sys.catalog]: This master's current role is: LEADER
I20260929 22:54:11.344722 2210 sys_catalog.cc:455] T 00000000000000000000000000000000 P c5174aff7fe34574aa07ea9e5dd38d69 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c5174aff7fe34574aa07ea9e5dd38d69. Latest consensus state: current_term: 1 leader_uuid: "c5174aff7fe34574aa07ea9e5dd38d69" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c5174aff7fe34574aa07ea9e5dd38d69" member_type: VOTER last_known_addr { host: "127.14.147.254" port: 43041 } } }
I20260929 22:54:11.344919 2210 sys_catalog.cc:458] T 00000000000000000000000000000000 P c5174aff7fe34574aa07ea9e5dd38d69 [sys.catalog]: This master's current role is: LEADER
I20260929 22:54:11.345077 2217 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 22:54:11.345677 2217 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 22:54:11.347183 2217 catalog_manager.cc:1347] Generated new cluster ID: 2c23378dec7143f8802008bdd252045c
I20260929 22:54:11.347230 2217 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 22:54:11.356714 2217 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 22:54:11.357220 2217 catalog_manager.cc:1504] Loading token signing keys...
I20260929 22:54:11.360630 2217 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c5174aff7fe34574aa07ea9e5dd38d69: Generated new TSK 0
I20260929 22:54:11.360816 2217 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 22:54:11.365226 14927 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhWOCLk/build/release/bin/kudu
/tmp/dist-test-taskhWOCLk/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790722213368738-14927-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790722213368738-14927-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790722213368738-14927-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790722213368738-14927-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.147.193:0
--local_ip_for_outbound_sockets=127.14.147.193
--webserver_interface=127.14.147.193
--webserver_port=0
--tserver_master_addrs=127.14.147.254:43041
--builtin_ntp_servers=127.14.147.212:40105
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260929 22:54:11.445243 2227 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 22:54:11.445465 2227 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 22:54:11.445497 2227 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 22:54:11.446736 2227 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 22:54:11.446784 2227 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 22:54:11.446822 2227 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.147.193
I20260929 22:54:11.448287 2227 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.147.212:40105
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790722213368738-14927-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790722213368738-14927-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.147.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790722213368738-14927-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.14.147.193
--webserver_port=0
--tserver_master_addrs=127.14.147.254:43041
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2227
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.147.193
--log_dir=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790722213368738-14927-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 21:58:54 UTC on ccycloud.quasar-xguoti.root.comops.site
build id 4681390
I20260929 22:54:11.448537 2227 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 22:54:11.448822 2227 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 22:54:11.454861 2233 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 22:54:11.454841 2232 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 22:54:11.454841 2235 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 22:54:11.455397 2227 server_base.cc:1034] running on GCE node
I20260929 22:54:11.455690 2227 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 22:54:11.456009 2227 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 22:54:11.457286 2227 hybrid_clock.cc:648] HybridClock initialized: now 1790722451457267 us; error 38 us; skew 500 ppm
I20260929 22:54:11.458678 2227 webserver.cc:460] Webserver started at http://127.14.147.193:35547/ using document root <none> and password file <none>
I20260929 22:54:11.458942 2227 fs_manager.cc:359] Metadata directory not provided
I20260929 22:54:11.458990 2227 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 22:54:11.459127 2227 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 22:54:11.460268 2227 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790722213368738-14927-0/minicluster-data/ts-0/data/instance:
uuid: "1442caba435a4bcc8a7030a781505d75"
format_stamp: "Formatted at 2026-09-29 22:54:11 on dist-test-slave-98mk"
server_key: "b07342e361c8da9cf2d9c017df6ae641"
server_key_iv: "1bb8ab86c6d33f84f53d1d760f48269b"
server_key_version: "encryptionkey@0"
I20260929 22:54:11.460609 2227 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790722213368738-14927-0/minicluster-data/ts-0/wal/instance:
uuid: "1442caba435a4bcc8a7030a781505d75"
format_stamp: "Formatted at 2026-09-29 22:54:11 on dist-test-slave-98mk"
server_key: "b07342e361c8da9cf2d9c017df6ae641"
server_key_iv: "1bb8ab86c6d33f84f53d1d760f48269b"
server_key_version: "encryptionkey@0"
I20260929 22:54:11.461874 2227 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260929 22:54:11.462641 2241 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 22:54:11.462831 2227 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260929 22:54:11.462914 2227 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790722213368738-14927-0/minicluster-data/ts-0/data,/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790722213368738-14927-0/minicluster-data/ts-0/wal
uuid: "1442caba435a4bcc8a7030a781505d75"
format_stamp: "Formatted at 2026-09-29 22:54:11 on dist-test-slave-98mk"
server_key: "b07342e361c8da9cf2d9c017df6ae641"
server_key_iv: "1bb8ab86c6d33f84f53d1d760f48269b"
server_key_version: "encryptionkey@0"
I20260929 22:54:11.462982 2227 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790722213368738-14927-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790722213368738-14927-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790722213368738-14927-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 22:54:11.476857 2227 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 22:54:11.477170 2227 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 22:54:11.477293 2227 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 22:54:11.477514 2227 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 22:54:11.477825 2227 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 22:54:11.477857 2227 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 22:54:11.477887 2227 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 22:54:11.477916 2227 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 22:54:11.483541 2227 rpc_server.cc:307] RPC server started. Bound to: 127.14.147.193:40521
I20260929 22:54:11.483596 2354 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.147.193:40521 every 8 connection(s)
I20260929 22:54:11.483937 2227 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790722213368738-14927-0/minicluster-data/ts-0/data/info.pb
I20260929 22:54:11.489022 2206 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:54:11.484654 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.147.193:48483 (local address 127.14.147.254:43041)
0929 22:54:11.484699 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0929 22:54:11.484701 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 22:54:11.484943 (+ 242us) server_negotiation.cc:407] Connection header received
0929 22:54:11.485453 (+ 510us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 22:54:11.485454 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 22:54:11.485473 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 22:54:11.485534 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0929 22:54:11.486199 (+ 665us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:54:11.486708 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:54:11.487687 (+ 979us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:54:11.487887 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:54:11.487949 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 22:54:11.488749 (+ 800us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 22:54:11.488750 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 22:54:11.488751 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 22:54:11.488756 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0929 22:54:11.488771 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 22:54:11.488775 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 22:54:11.488820 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0929 22:54:11.488904 (+ 84us) server_negotiation.cc:299] Negotiation successful
0929 22:54:11.488939 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260929 22:54:11.489212 2357 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:54:11.484629 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.147.254:43041 (local address 127.14.147.193:48483)
0929 22:54:11.484810 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0929 22:54:11.484843 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0929 22:54:11.485347 (+ 504us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 22:54:11.485600 (+ 253us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 22:54:11.485604 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 22:54:11.485612 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0929 22:54:11.486106 (+ 494us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 22:54:11.486110 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 22:54:11.486798 (+ 688us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 22:54:11.486799 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 22:54:11.487595 (+ 796us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 22:54:11.487597 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 22:54:11.487961 (+ 364us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 22:54:11.487962 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 22:54:11.488085 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 22:54:11.488609 (+ 524us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 22:54:11.488609 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0929 22:54:11.488638 (+ 29us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 22:54:11.488662 (+ 24us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 22:54:11.488827 (+ 165us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 22:54:11.488833 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 22:54:11.488836 (+ 3us) client_negotiation.cc:769] Sending connection context
0929 22:54:11.488896 (+ 60us) client_negotiation.cc:240] Negotiation successful
0929 22:54:11.489002 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":89,"threads_started":1}
I20260929 22:54:11.489686 2355 heartbeater.cc:347] Connected to a master server at 127.14.147.254:43041
I20260929 22:54:11.489768 2355 heartbeater.cc:464] Registering TS with master...
I20260929 22:54:11.489959 2355 heartbeater.cc:511] Master 127.14.147.254:43041 requested a full tablet report, sending...
I20260929 22:54:11.490098 14927 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhWOCLk/build/release/bin/kudu as pid 2227
I20260929 22:54:11.490198 14927 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790722213368738-14927-0/minicluster-data/ts-0/wal/instance
I20260929 22:54:11.490406 14927 external_mini_cluster.cc:1428] Setting key 9a5968c94be2f0b6d8f3ea3df540cc6b
I20260929 22:54:11.490476 2167 ts_manager.cc:194] Registered new tserver with Master: 1442caba435a4bcc8a7030a781505d75 (127.14.147.193:40521)
I20260929 22:54:11.491336 2167 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.14.147.193:48483
I20260929 22:54:11.491477 14927 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhWOCLk/build/release/bin/kudu
/tmp/dist-test-taskhWOCLk/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790722213368738-14927-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790722213368738-14927-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790722213368738-14927-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790722213368738-14927-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.147.194:0
--local_ip_for_outbound_sockets=127.14.147.194
--webserver_interface=127.14.147.194
--webserver_port=0
--tserver_master_addrs=127.14.147.254:43041
--builtin_ntp_servers=127.14.147.212:40105
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260929 22:54:11.574355 2358 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 22:54:11.574582 2358 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 22:54:11.574606 2358 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 22:54:11.575870 2358 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 22:54:11.575918 2358 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 22:54:11.575946 2358 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.147.194
I20260929 22:54:11.577378 2358 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.147.212:40105
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790722213368738-14927-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790722213368738-14927-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.147.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790722213368738-14927-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.14.147.194
--webserver_port=0
--tserver_master_addrs=127.14.147.254:43041
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2358
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.147.194
--log_dir=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790722213368738-14927-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 21:58:54 UTC on ccycloud.quasar-xguoti.root.comops.site
build id 4681390
I20260929 22:54:11.577632 2358 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 22:54:11.577931 2358 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 22:54:11.582823 2364 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 22:54:11.582796 2363 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 22:54:11.583051 2358 server_base.cc:1034] running on GCE node
W20260929 22:54:11.582804 2366 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 22:54:11.583268 2358 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 22:54:11.583540 2358 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 22:54:11.584692 2358 hybrid_clock.cc:648] HybridClock initialized: now 1790722451584673 us; error 33 us; skew 500 ppm
I20260929 22:54:11.586031 2358 webserver.cc:460] Webserver started at http://127.14.147.194:37547/ using document root <none> and password file <none>
I20260929 22:54:11.586293 2358 fs_manager.cc:359] Metadata directory not provided
I20260929 22:54:11.586345 2358 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 22:54:11.586479 2358 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 22:54:11.587654 2358 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790722213368738-14927-0/minicluster-data/ts-1/data/instance:
uuid: "c8f0afee5cf34e398dcdd5ac051df6af"
format_stamp: "Formatted at 2026-09-29 22:54:11 on dist-test-slave-98mk"
server_key: "72c9c84997e74c5effd7a40877dd5630"
server_key_iv: "f65c50ac9e68b666a6ab8e28bb01f4cc"
server_key_version: "encryptionkey@0"
I20260929 22:54:11.587998 2358 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790722213368738-14927-0/minicluster-data/ts-1/wal/instance:
uuid: "c8f0afee5cf34e398dcdd5ac051df6af"
format_stamp: "Formatted at 2026-09-29 22:54:11 on dist-test-slave-98mk"
server_key: "72c9c84997e74c5effd7a40877dd5630"
server_key_iv: "f65c50ac9e68b666a6ab8e28bb01f4cc"
server_key_version: "encryptionkey@0"
I20260929 22:54:11.589298 2358 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260929 22:54:11.590080 2372 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 22:54:11.590286 2358 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260929 22:54:11.590380 2358 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790722213368738-14927-0/minicluster-data/ts-1/data,/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790722213368738-14927-0/minicluster-data/ts-1/wal
uuid: "c8f0afee5cf34e398dcdd5ac051df6af"
format_stamp: "Formatted at 2026-09-29 22:54:11 on dist-test-slave-98mk"
server_key: "72c9c84997e74c5effd7a40877dd5630"
server_key_iv: "f65c50ac9e68b666a6ab8e28bb01f4cc"
server_key_version: "encryptionkey@0"
I20260929 22:54:11.590459 2358 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790722213368738-14927-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790722213368738-14927-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790722213368738-14927-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 22:54:11.598424 2358 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 22:54:11.598729 2358 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 22:54:11.598860 2358 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 22:54:11.599072 2358 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 22:54:11.599407 2358 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 22:54:11.599498 2358 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 22:54:11.599530 2358 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 22:54:11.599555 2358 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 22:54:11.604998 2358 rpc_server.cc:307] RPC server started. Bound to: 127.14.147.194:45741
I20260929 22:54:11.605069 2485 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.147.194:45741 every 8 connection(s)
I20260929 22:54:11.605376 2358 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790722213368738-14927-0/minicluster-data/ts-1/data/info.pb
I20260929 22:54:11.606273 14927 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhWOCLk/build/release/bin/kudu as pid 2358
I20260929 22:54:11.606369 14927 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790722213368738-14927-0/minicluster-data/ts-1/wal/instance
I20260929 22:54:11.606550 14927 external_mini_cluster.cc:1428] Setting key 58e3e263bdcd6674d5fd8e225df77c1a
I20260929 22:54:11.607487 14927 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhWOCLk/build/release/bin/kudu
/tmp/dist-test-taskhWOCLk/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790722213368738-14927-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790722213368738-14927-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790722213368738-14927-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790722213368738-14927-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.147.195:0
--local_ip_for_outbound_sockets=127.14.147.195
--webserver_interface=127.14.147.195
--webserver_port=0
--tserver_master_addrs=127.14.147.254:43041
--builtin_ntp_servers=127.14.147.212:40105
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260929 22:54:11.610611 2206 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:54:11.606068 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.147.194:53045 (local address 127.14.147.254:43041)
0929 22:54:11.606147 (+ 79us) server_negotiation.cc:206] Beginning negotiation
0929 22:54:11.606150 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 22:54:11.606371 (+ 221us) server_negotiation.cc:407] Connection header received
0929 22:54:11.606966 (+ 595us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 22:54:11.606967 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 22:54:11.606979 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 22:54:11.607021 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0929 22:54:11.607838 (+ 817us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:54:11.608335 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:54:11.609296 (+ 961us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:54:11.609495 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:54:11.609548 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 22:54:11.610355 (+ 807us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 22:54:11.610356 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 22:54:11.610356 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 22:54:11.610361 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0929 22:54:11.610377 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 22:54:11.610381 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 22:54:11.610425 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0929 22:54:11.610505 (+ 80us) server_negotiation.cc:299] Negotiation successful
0929 22:54:11.610531 (+ 26us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260929 22:54:11.610818 2488 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:54:11.606057 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.147.254:43041 (local address 127.14.147.194:53045)
0929 22:54:11.606244 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0929 22:54:11.606285 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0929 22:54:11.606872 (+ 587us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 22:54:11.607105 (+ 233us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 22:54:11.607110 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 22:54:11.607120 (+ 10us) client_negotiation.cc:189] Negotiated authn=SASL
0929 22:54:11.607753 (+ 633us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 22:54:11.607759 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 22:54:11.608424 (+ 665us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 22:54:11.608425 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 22:54:11.609213 (+ 788us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 22:54:11.609216 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 22:54:11.609564 (+ 348us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 22:54:11.609565 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 22:54:11.609708 (+ 143us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 22:54:11.610223 (+ 515us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 22:54:11.610223 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0929 22:54:11.610252 (+ 29us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 22:54:11.610275 (+ 23us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 22:54:11.610431 (+ 156us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 22:54:11.610443 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 22:54:11.610446 (+ 3us) client_negotiation.cc:769] Sending connection context
0929 22:54:11.610507 (+ 61us) client_negotiation.cc:240] Negotiation successful
0929 22:54:11.610613 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":78,"threads_started":1}
I20260929 22:54:11.611267 2486 heartbeater.cc:347] Connected to a master server at 127.14.147.254:43041
I20260929 22:54:11.611351 2486 heartbeater.cc:464] Registering TS with master...
I20260929 22:54:11.611560 2486 heartbeater.cc:511] Master 127.14.147.254:43041 requested a full tablet report, sending...
I20260929 22:54:11.611920 2167 ts_manager.cc:194] Registered new tserver with Master: c8f0afee5cf34e398dcdd5ac051df6af (127.14.147.194:45741)
I20260929 22:54:11.612414 2167 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.14.147.194:53045
W20260929 22:54:11.688004 2489 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 22:54:11.688212 2489 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 22:54:11.688234 2489 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 22:54:11.689376 2489 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 22:54:11.689412 2489 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 22:54:11.689438 2489 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.147.195
I20260929 22:54:11.690752 2489 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.147.212:40105
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790722213368738-14927-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790722213368738-14927-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.147.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790722213368738-14927-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.14.147.195
--webserver_port=0
--tserver_master_addrs=127.14.147.254:43041
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2489
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.147.195
--log_dir=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790722213368738-14927-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 21:58:54 UTC on ccycloud.quasar-xguoti.root.comops.site
build id 4681390
I20260929 22:54:11.690964 2489 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 22:54:11.691226 2489 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 22:54:11.696115 2495 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 22:54:11.696121 2494 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 22:54:11.696121 2497 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 22:54:11.696319 2489 server_base.cc:1034] running on GCE node
I20260929 22:54:11.696564 2489 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 22:54:11.696780 2489 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 22:54:11.697924 2489 hybrid_clock.cc:648] HybridClock initialized: now 1790722451697913 us; error 33 us; skew 500 ppm
I20260929 22:54:11.699213 2489 webserver.cc:460] Webserver started at http://127.14.147.195:40925/ using document root <none> and password file <none>
I20260929 22:54:11.699548 2489 fs_manager.cc:359] Metadata directory not provided
I20260929 22:54:11.699609 2489 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 22:54:11.699743 2489 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 22:54:11.700893 2489 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790722213368738-14927-0/minicluster-data/ts-2/data/instance:
uuid: "f8e06c7affda4c13b29a9dc716c34d96"
format_stamp: "Formatted at 2026-09-29 22:54:11 on dist-test-slave-98mk"
server_key: "3adec523a61e7befbfd823ab65a02e84"
server_key_iv: "234c9c2877b32c2b48c70e80a87bb7e0"
server_key_version: "encryptionkey@0"
I20260929 22:54:11.701213 2489 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790722213368738-14927-0/minicluster-data/ts-2/wal/instance:
uuid: "f8e06c7affda4c13b29a9dc716c34d96"
format_stamp: "Formatted at 2026-09-29 22:54:11 on dist-test-slave-98mk"
server_key: "3adec523a61e7befbfd823ab65a02e84"
server_key_iv: "234c9c2877b32c2b48c70e80a87bb7e0"
server_key_version: "encryptionkey@0"
I20260929 22:54:11.702471 2489 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260929 22:54:11.703235 2503 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 22:54:11.703451 2489 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260929 22:54:11.703542 2489 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790722213368738-14927-0/minicluster-data/ts-2/data,/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790722213368738-14927-0/minicluster-data/ts-2/wal
uuid: "f8e06c7affda4c13b29a9dc716c34d96"
format_stamp: "Formatted at 2026-09-29 22:54:11 on dist-test-slave-98mk"
server_key: "3adec523a61e7befbfd823ab65a02e84"
server_key_iv: "234c9c2877b32c2b48c70e80a87bb7e0"
server_key_version: "encryptionkey@0"
I20260929 22:54:11.703613 2489 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790722213368738-14927-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790722213368738-14927-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790722213368738-14927-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 22:54:11.728349 2489 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 22:54:11.728669 2489 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 22:54:11.728807 2489 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 22:54:11.729029 2489 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 22:54:11.729363 2489 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 22:54:11.729398 2489 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 22:54:11.729427 2489 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 22:54:11.729451 2489 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 22:54:11.735039 2489 rpc_server.cc:307] RPC server started. Bound to: 127.14.147.195:34193
I20260929 22:54:11.735291 2616 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.147.195:34193 every 8 connection(s)
I20260929 22:54:11.735491 2489 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790722213368738-14927-0/minicluster-data/ts-2/data/info.pb
I20260929 22:54:11.740777 2206 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:54:11.736374 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.147.195:59787 (local address 127.14.147.254:43041)
0929 22:54:11.736453 (+ 79us) server_negotiation.cc:206] Beginning negotiation
0929 22:54:11.736456 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 22:54:11.736627 (+ 171us) server_negotiation.cc:407] Connection header received
0929 22:54:11.737155 (+ 528us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 22:54:11.737157 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 22:54:11.737167 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 22:54:11.737225 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0929 22:54:11.737872 (+ 647us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:54:11.738382 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:54:11.739452 (+ 1070us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:54:11.739689 (+ 237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:54:11.739748 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 22:54:11.740593 (+ 845us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 22:54:11.740594 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 22:54:11.740594 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 22:54:11.740598 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0929 22:54:11.740607 (+ 9us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 22:54:11.740610 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 22:54:11.740641 (+ 31us) server_negotiation.cc:1035] Waiting for connection context
0929 22:54:11.740694 (+ 53us) server_negotiation.cc:299] Negotiation successful
0929 22:54:11.740713 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260929 22:54:11.740970 2619 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:54:11.736306 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.147.254:43041 (local address 127.14.147.195:59787)
0929 22:54:11.736507 (+ 201us) negotiation.cc:107] Waiting for socket to connect
0929 22:54:11.736547 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0929 22:54:11.737036 (+ 489us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 22:54:11.737278 (+ 242us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 22:54:11.737282 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 22:54:11.737290 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0929 22:54:11.737791 (+ 501us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 22:54:11.737795 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 22:54:11.738490 (+ 695us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 22:54:11.738491 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 22:54:11.739320 (+ 829us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 22:54:11.739323 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 22:54:11.739792 (+ 469us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 22:54:11.739793 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 22:54:11.739922 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 22:54:11.740475 (+ 553us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 22:54:11.740476 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0929 22:54:11.740505 (+ 29us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 22:54:11.740530 (+ 25us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 22:54:11.740644 (+ 114us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 22:54:11.740646 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 22:54:11.740648 (+ 2us) client_negotiation.cc:769] Sending connection context
0929 22:54:11.740692 (+ 44us) client_negotiation.cc:240] Negotiation successful
0929 22:54:11.740773 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":134,"thread_start_us":81,"threads_started":1}
I20260929 22:54:11.741402 2617 heartbeater.cc:347] Connected to a master server at 127.14.147.254:43041
I20260929 22:54:11.741498 2617 heartbeater.cc:464] Registering TS with master...
I20260929 22:54:11.741649 2617 heartbeater.cc:511] Master 127.14.147.254:43041 requested a full tablet report, sending...
I20260929 22:54:11.741982 2167 ts_manager.cc:194] Registered new tserver with Master: f8e06c7affda4c13b29a9dc716c34d96 (127.14.147.195:34193)
I20260929 22:54:11.742326 2167 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.14.147.195:59787
I20260929 22:54:11.742338 14927 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhWOCLk/build/release/bin/kudu as pid 2489
I20260929 22:54:11.742442 14927 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790722213368738-14927-0/minicluster-data/ts-2/wal/instance
I20260929 22:54:11.742647 14927 external_mini_cluster.cc:1428] Setting key 10f4ef098c3451c595f209814f8a04ae
I20260929 22:54:11.743466 14927 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 22:54:11.747707 2206 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:54:11.744791 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59274 (local address 127.14.147.254:43041)
0929 22:54:11.744830 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0929 22:54:11.744832 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 22:54:11.744956 (+ 124us) server_negotiation.cc:407] Connection header received
0929 22:54:11.745024 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 22:54:11.745024 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 22:54:11.745036 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 22:54:11.745080 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0929 22:54:11.745229 (+ 149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:54:11.745809 (+ 580us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:54:11.746560 (+ 751us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:54:11.746760 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:54:11.746815 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 22:54:11.747470 (+ 655us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 22:54:11.747471 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 22:54:11.747472 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 22:54:11.747477 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0929 22:54:11.747492 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 22:54:11.747496 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 22:54:11.747541 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0929 22:54:11.747600 (+ 59us) server_negotiation.cc:299] Negotiation successful
0929 22:54:11.747624 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260929 22:54:11.750100 2167 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:59274:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 22:54:11.751849 2167 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 22:54:11.758389 2628 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:54:11.754131 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58146 (local address 127.14.147.195:34193)
0929 22:54:11.754639 (+ 508us) server_negotiation.cc:206] Beginning negotiation
0929 22:54:11.754644 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 22:54:11.754653 (+ 9us) server_negotiation.cc:407] Connection header received
0929 22:54:11.754697 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 22:54:11.754700 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 22:54:11.754714 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 22:54:11.754786 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 22:54:11.754887 (+ 101us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:54:11.755736 (+ 849us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:54:11.757364 (+ 1628us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:54:11.757947 (+ 583us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:54:11.758026 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 22:54:11.758122 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0929 22:54:11.758252 (+ 130us) server_negotiation.cc:299] Negotiation successful
0929 22:54:11.758291 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":472,"thread_start_us":79,"threads_started":1}
I20260929 22:54:11.758381 2625 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:54:11.753948 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.147.195:34193 (local address 127.0.0.1:58146)
0929 22:54:11.754130 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0929 22:54:11.754171 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0929 22:54:11.754235 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 22:54:11.754785 (+ 550us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 22:54:11.754786 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 22:54:11.754789 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 22:54:11.754825 (+ 36us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 22:54:11.754827 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 22:54:11.755818 (+ 991us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 22:54:11.755820 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 22:54:11.757242 (+ 1422us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 22:54:11.757243 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 22:54:11.758040 (+ 797us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 22:54:11.758041 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 22:54:11.758130 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 22:54:11.758145 (+ 15us) client_negotiation.cc:769] Sending connection context
0929 22:54:11.758227 (+ 82us) client_negotiation.cc:240] Negotiation successful
0929 22:54:11.758293 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":157,"mutex_wait_us":9,"thread_start_us":74,"threads_started":1}
I20260929 22:54:11.758571 2627 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:54:11.753951 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.147.194:45741 (local address 127.0.0.1:53822)
0929 22:54:11.754130 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0929 22:54:11.754171 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0929 22:54:11.754235 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 22:54:11.754557 (+ 322us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 22:54:11.754560 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 22:54:11.754562 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 22:54:11.754643 (+ 81us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 22:54:11.754645 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 22:54:11.756118 (+ 1473us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 22:54:11.756119 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 22:54:11.757236 (+ 1117us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 22:54:11.757241 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 22:54:11.758364 (+ 1123us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 22:54:11.758365 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 22:54:11.758418 (+ 53us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 22:54:11.758422 (+ 4us) client_negotiation.cc:769] Sending connection context
0929 22:54:11.758470 (+ 48us) client_negotiation.cc:240] Negotiation successful
0929 22:54:11.758511 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":156,"mutex_wait_us":3,"thread_start_us":110,"threads_started":1}
I20260929 22:54:11.758837 2629 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:54:11.754192 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53822 (local address 127.14.147.194:45741)
0929 22:54:11.754360 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0929 22:54:11.754366 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 22:54:11.754377 (+ 11us) server_negotiation.cc:407] Connection header received
0929 22:54:11.754423 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 22:54:11.754425 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 22:54:11.754442 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 22:54:11.754533 (+ 91us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 22:54:11.754727 (+ 194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:54:11.755991 (+ 1264us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:54:11.757382 (+ 1391us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:54:11.758171 (+ 789us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:54:11.758247 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 22:54:11.758329 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0929 22:54:11.758732 (+ 403us) server_negotiation.cc:299] Negotiation successful
0929 22:54:11.758774 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":128,"thread_start_us":63,"threads_started":1}
I20260929 22:54:11.758960 2626 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:54:11.753956 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.147.193:40521 (local address 127.0.0.1:35648)
0929 22:54:11.754329 (+ 373us) negotiation.cc:107] Waiting for socket to connect
0929 22:54:11.754336 (+ 7us) client_negotiation.cc:174] Beginning negotiation
0929 22:54:11.754369 (+ 33us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 22:54:11.755025 (+ 656us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 22:54:11.755026 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 22:54:11.755029 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 22:54:11.755072 (+ 43us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 22:54:11.755074 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 22:54:11.756478 (+ 1404us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 22:54:11.756480 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 22:54:11.757682 (+ 1202us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 22:54:11.757686 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 22:54:11.758710 (+ 1024us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 22:54:11.758711 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 22:54:11.758789 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 22:54:11.758793 (+ 4us) client_negotiation.cc:769] Sending connection context
0929 22:54:11.758839 (+ 46us) client_negotiation.cc:240] Negotiation successful
0929 22:54:11.758885 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":361,"spinlock_wait_cycles":768,"thread_start_us":83,"threads_started":1}
I20260929 22:54:11.759238 2630 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:54:11.754297 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35648 (local address 127.14.147.193:40521)
0929 22:54:11.754499 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0929 22:54:11.754503 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 22:54:11.754512 (+ 9us) server_negotiation.cc:407] Connection header received
0929 22:54:11.754556 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 22:54:11.754558 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 22:54:11.754572 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 22:54:11.754657 (+ 85us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 22:54:11.755153 (+ 496us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:54:11.756373 (+ 1220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:54:11.757800 (+ 1427us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:54:11.758524 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:54:11.758597 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 22:54:11.758675 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0929 22:54:11.759112 (+ 437us) server_negotiation.cc:299] Negotiation successful
0929 22:54:11.759161 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":159,"thread_start_us":89,"threads_started":1}
I20260929 22:54:11.759583 2551 tablet_service.cc:1467] Processing CreateTablet for tablet fe4fdcedc6094fb6afeaebef7c533bcb (DEFAULT_TABLE table=test-table [id=a4abccf85ef24c7eaab888f5f017a7ee]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 22:54:11.759608 2420 tablet_service.cc:1467] Processing CreateTablet for tablet fe4fdcedc6094fb6afeaebef7c533bcb (DEFAULT_TABLE table=test-table [id=a4abccf85ef24c7eaab888f5f017a7ee]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 22:54:11.759963 2420 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fe4fdcedc6094fb6afeaebef7c533bcb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 22:54:11.759963 2551 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fe4fdcedc6094fb6afeaebef7c533bcb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 22:54:11.762611 2632 tablet_bootstrap.cc:492] T fe4fdcedc6094fb6afeaebef7c533bcb P f8e06c7affda4c13b29a9dc716c34d96: Bootstrap starting.
I20260929 22:54:11.763265 2632 tablet_bootstrap.cc:654] T fe4fdcedc6094fb6afeaebef7c533bcb P f8e06c7affda4c13b29a9dc716c34d96: Neither blocks nor log segments found. Creating new log.
I20260929 22:54:11.763229 2289 tablet_service.cc:1467] Processing CreateTablet for tablet fe4fdcedc6094fb6afeaebef7c533bcb (DEFAULT_TABLE table=test-table [id=a4abccf85ef24c7eaab888f5f017a7ee]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 22:54:11.763551 2289 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fe4fdcedc6094fb6afeaebef7c533bcb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 22:54:11.763593 2631 tablet_bootstrap.cc:492] T fe4fdcedc6094fb6afeaebef7c533bcb P c8f0afee5cf34e398dcdd5ac051df6af: Bootstrap starting.
I20260929 22:54:11.763624 2632 log.cc:824] T fe4fdcedc6094fb6afeaebef7c533bcb P f8e06c7affda4c13b29a9dc716c34d96: Log is configured to *not* fsync() on all Append() calls
I20260929 22:54:11.764204 2631 tablet_bootstrap.cc:654] T fe4fdcedc6094fb6afeaebef7c533bcb P c8f0afee5cf34e398dcdd5ac051df6af: Neither blocks nor log segments found. Creating new log.
I20260929 22:54:11.764400 2632 tablet_bootstrap.cc:492] T fe4fdcedc6094fb6afeaebef7c533bcb P f8e06c7affda4c13b29a9dc716c34d96: No bootstrap required, opened a new log
I20260929 22:54:11.764482 2632 ts_tablet_manager.cc:1397] T fe4fdcedc6094fb6afeaebef7c533bcb P f8e06c7affda4c13b29a9dc716c34d96: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260929 22:54:11.764528 2631 log.cc:824] T fe4fdcedc6094fb6afeaebef7c533bcb P c8f0afee5cf34e398dcdd5ac051df6af: Log is configured to *not* fsync() on all Append() calls
I20260929 22:54:11.765406 2631 tablet_bootstrap.cc:492] T fe4fdcedc6094fb6afeaebef7c533bcb P c8f0afee5cf34e398dcdd5ac051df6af: No bootstrap required, opened a new log
I20260929 22:54:11.765481 2631 ts_tablet_manager.cc:1397] T fe4fdcedc6094fb6afeaebef7c533bcb P c8f0afee5cf34e398dcdd5ac051df6af: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260929 22:54:11.766846 2635 tablet_bootstrap.cc:492] T fe4fdcedc6094fb6afeaebef7c533bcb P 1442caba435a4bcc8a7030a781505d75: Bootstrap starting.
I20260929 22:54:11.767053 2632 raft_consensus.cc:348] T fe4fdcedc6094fb6afeaebef7c533bcb P f8e06c7affda4c13b29a9dc716c34d96 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1442caba435a4bcc8a7030a781505d75" member_type: VOTER last_known_addr { host: "127.14.147.193" port: 40521 } } peers { permanent_uuid: "f8e06c7affda4c13b29a9dc716c34d96" member_type: VOTER last_known_addr { host: "127.14.147.195" port: 34193 } } peers { permanent_uuid: "c8f0afee5cf34e398dcdd5ac051df6af" member_type: VOTER last_known_addr { host: "127.14.147.194" port: 45741 } }
I20260929 22:54:11.767238 2632 raft_consensus.cc:374] T fe4fdcedc6094fb6afeaebef7c533bcb P f8e06c7affda4c13b29a9dc716c34d96 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 22:54:11.767273 2632 raft_consensus.cc:729] T fe4fdcedc6094fb6afeaebef7c533bcb P f8e06c7affda4c13b29a9dc716c34d96 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f8e06c7affda4c13b29a9dc716c34d96, State: Initialized, Role: FOLLOWER
I20260929 22:54:11.767459 2632 consensus_queue.cc:260] T fe4fdcedc6094fb6afeaebef7c533bcb P f8e06c7affda4c13b29a9dc716c34d96 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1442caba435a4bcc8a7030a781505d75" member_type: VOTER last_known_addr { host: "127.14.147.193" port: 40521 } } peers { permanent_uuid: "f8e06c7affda4c13b29a9dc716c34d96" member_type: VOTER last_known_addr { host: "127.14.147.195" port: 34193 } } peers { permanent_uuid: "c8f0afee5cf34e398dcdd5ac051df6af" member_type: VOTER last_known_addr { host: "127.14.147.194" port: 45741 } }
I20260929 22:54:11.767545 2635 tablet_bootstrap.cc:654] T fe4fdcedc6094fb6afeaebef7c533bcb P 1442caba435a4bcc8a7030a781505d75: Neither blocks nor log segments found. Creating new log.
I20260929 22:54:11.767742 2632 ts_tablet_manager.cc:1428] T fe4fdcedc6094fb6afeaebef7c533bcb P f8e06c7affda4c13b29a9dc716c34d96: Time spent starting tablet: real 0.003s user 0.002s sys 0.001s
I20260929 22:54:11.767838 2635 log.cc:824] T fe4fdcedc6094fb6afeaebef7c533bcb P 1442caba435a4bcc8a7030a781505d75: Log is configured to *not* fsync() on all Append() calls
I20260929 22:54:11.767899 2617 heartbeater.cc:503] Master 127.14.147.254:43041 was elected leader, sending a full tablet report...
I20260929 22:54:11.768036 2631 raft_consensus.cc:348] T fe4fdcedc6094fb6afeaebef7c533bcb P c8f0afee5cf34e398dcdd5ac051df6af [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1442caba435a4bcc8a7030a781505d75" member_type: VOTER last_known_addr { host: "127.14.147.193" port: 40521 } } peers { permanent_uuid: "f8e06c7affda4c13b29a9dc716c34d96" member_type: VOTER last_known_addr { host: "127.14.147.195" port: 34193 } } peers { permanent_uuid: "c8f0afee5cf34e398dcdd5ac051df6af" member_type: VOTER last_known_addr { host: "127.14.147.194" port: 45741 } }
I20260929 22:54:11.768199 2631 raft_consensus.cc:374] T fe4fdcedc6094fb6afeaebef7c533bcb P c8f0afee5cf34e398dcdd5ac051df6af [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 22:54:11.768246 2631 raft_consensus.cc:729] T fe4fdcedc6094fb6afeaebef7c533bcb P c8f0afee5cf34e398dcdd5ac051df6af [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c8f0afee5cf34e398dcdd5ac051df6af, State: Initialized, Role: FOLLOWER
I20260929 22:54:11.768368 2631 consensus_queue.cc:260] T fe4fdcedc6094fb6afeaebef7c533bcb P c8f0afee5cf34e398dcdd5ac051df6af [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1442caba435a4bcc8a7030a781505d75" member_type: VOTER last_known_addr { host: "127.14.147.193" port: 40521 } } peers { permanent_uuid: "f8e06c7affda4c13b29a9dc716c34d96" member_type: VOTER last_known_addr { host: "127.14.147.195" port: 34193 } } peers { permanent_uuid: "c8f0afee5cf34e398dcdd5ac051df6af" member_type: VOTER last_known_addr { host: "127.14.147.194" port: 45741 } }
I20260929 22:54:11.768625 2635 tablet_bootstrap.cc:492] T fe4fdcedc6094fb6afeaebef7c533bcb P 1442caba435a4bcc8a7030a781505d75: No bootstrap required, opened a new log
I20260929 22:54:11.768623 2631 ts_tablet_manager.cc:1428] T fe4fdcedc6094fb6afeaebef7c533bcb P c8f0afee5cf34e398dcdd5ac051df6af: Time spent starting tablet: real 0.003s user 0.000s sys 0.004s
I20260929 22:54:11.768683 2635 ts_tablet_manager.cc:1397] T fe4fdcedc6094fb6afeaebef7c533bcb P 1442caba435a4bcc8a7030a781505d75: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260929 22:54:11.768671 2486 heartbeater.cc:503] Master 127.14.147.254:43041 was elected leader, sending a full tablet report...
I20260929 22:54:11.770735 2635 raft_consensus.cc:348] T fe4fdcedc6094fb6afeaebef7c533bcb P 1442caba435a4bcc8a7030a781505d75 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1442caba435a4bcc8a7030a781505d75" member_type: VOTER last_known_addr { host: "127.14.147.193" port: 40521 } } peers { permanent_uuid: "f8e06c7affda4c13b29a9dc716c34d96" member_type: VOTER last_known_addr { host: "127.14.147.195" port: 34193 } } peers { permanent_uuid: "c8f0afee5cf34e398dcdd5ac051df6af" member_type: VOTER last_known_addr { host: "127.14.147.194" port: 45741 } }
I20260929 22:54:11.770864 2635 raft_consensus.cc:374] T fe4fdcedc6094fb6afeaebef7c533bcb P 1442caba435a4bcc8a7030a781505d75 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 22:54:11.770884 2635 raft_consensus.cc:729] T fe4fdcedc6094fb6afeaebef7c533bcb P 1442caba435a4bcc8a7030a781505d75 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1442caba435a4bcc8a7030a781505d75, State: Initialized, Role: FOLLOWER
I20260929 22:54:11.770990 2635 consensus_queue.cc:260] T fe4fdcedc6094fb6afeaebef7c533bcb P 1442caba435a4bcc8a7030a781505d75 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1442caba435a4bcc8a7030a781505d75" member_type: VOTER last_known_addr { host: "127.14.147.193" port: 40521 } } peers { permanent_uuid: "f8e06c7affda4c13b29a9dc716c34d96" member_type: VOTER last_known_addr { host: "127.14.147.195" port: 34193 } } peers { permanent_uuid: "c8f0afee5cf34e398dcdd5ac051df6af" member_type: VOTER last_known_addr { host: "127.14.147.194" port: 45741 } }
I20260929 22:54:11.771217 2635 ts_tablet_manager.cc:1428] T fe4fdcedc6094fb6afeaebef7c533bcb P 1442caba435a4bcc8a7030a781505d75: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260929 22:54:11.771278 2355 heartbeater.cc:503] Master 127.14.147.254:43041 was elected leader, sending a full tablet report...
W20260929 22:54:11.855933 2487 tablet.cc:2369] T fe4fdcedc6094fb6afeaebef7c533bcb P c8f0afee5cf34e398dcdd5ac051df6af: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260929 22:54:11.984628 2356 tablet.cc:2369] T fe4fdcedc6094fb6afeaebef7c533bcb P 1442caba435a4bcc8a7030a781505d75: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260929 22:54:11.986069 2618 tablet.cc:2369] T fe4fdcedc6094fb6afeaebef7c533bcb P f8e06c7affda4c13b29a9dc716c34d96: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 22:54:12.161556 2638 raft_consensus.cc:482] T fe4fdcedc6094fb6afeaebef7c533bcb P c8f0afee5cf34e398dcdd5ac051df6af [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 22:54:12.161705 2638 raft_consensus.cc:504] T fe4fdcedc6094fb6afeaebef7c533bcb P c8f0afee5cf34e398dcdd5ac051df6af [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1442caba435a4bcc8a7030a781505d75" member_type: VOTER last_known_addr { host: "127.14.147.193" port: 40521 } } peers { permanent_uuid: "f8e06c7affda4c13b29a9dc716c34d96" member_type: VOTER last_known_addr { host: "127.14.147.195" port: 34193 } } peers { permanent_uuid: "c8f0afee5cf34e398dcdd5ac051df6af" member_type: VOTER last_known_addr { host: "127.14.147.194" port: 45741 } }
I20260929 22:54:12.162071 2638 leader_election.cc:288] T fe4fdcedc6094fb6afeaebef7c533bcb P c8f0afee5cf34e398dcdd5ac051df6af [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1442caba435a4bcc8a7030a781505d75 (127.14.147.193:40521), f8e06c7affda4c13b29a9dc716c34d96 (127.14.147.195:34193)
I20260929 22:54:12.164959 2628 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:54:12.162322 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.147.194:37875 (local address 127.14.147.195:34193)
0929 22:54:12.162394 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0929 22:54:12.162396 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 22:54:12.162459 (+ 63us) server_negotiation.cc:407] Connection header received
0929 22:54:12.162528 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 22:54:12.162529 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 22:54:12.162541 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 22:54:12.162589 (+ 48us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 22:54:12.162759 (+ 170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:54:12.163247 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:54:12.164278 (+ 1031us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:54:12.164634 (+ 356us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:54:12.164701 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 22:54:12.164711 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0929 22:54:12.164850 (+ 139us) server_negotiation.cc:299] Negotiation successful
0929 22:54:12.164894 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260929 22:54:12.165004 2630 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:54:12.162400 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.147.194:57957 (local address 127.14.147.193:40521)
0929 22:54:12.162469 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0929 22:54:12.162472 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 22:54:12.162531 (+ 59us) server_negotiation.cc:407] Connection header received
0929 22:54:12.162592 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 22:54:12.162593 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 22:54:12.162604 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 22:54:12.162654 (+ 50us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 22:54:12.162801 (+ 147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:54:12.163251 (+ 450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:54:12.164278 (+ 1027us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:54:12.164638 (+ 360us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:54:12.164705 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 22:54:12.164713 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0929 22:54:12.164858 (+ 145us) server_negotiation.cc:299] Negotiation successful
0929 22:54:12.164939 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260929 22:54:12.165017 2640 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:54:12.162255 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.147.195:34193 (local address 127.14.147.194:37875)
0929 22:54:12.162390 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0929 22:54:12.162400 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0929 22:54:12.162470 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 22:54:12.162604 (+ 134us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 22:54:12.162605 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 22:54:12.162607 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 22:54:12.162693 (+ 86us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 22:54:12.162696 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 22:54:12.163332 (+ 636us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 22:54:12.163333 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 22:54:12.164188 (+ 855us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 22:54:12.164191 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 22:54:12.164713 (+ 522us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 22:54:12.164714 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 22:54:12.164779 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 22:54:12.164783 (+ 4us) client_negotiation.cc:769] Sending connection context
0929 22:54:12.164822 (+ 39us) client_negotiation.cc:240] Negotiation successful
0929 22:54:12.164867 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":120,"mutex_wait_us":34,"thread_start_us":69,"threads_started":1}
I20260929 22:54:12.165172 2641 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:54:12.162254 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.147.193:40521 (local address 127.14.147.194:57957)
0929 22:54:12.162444 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0929 22:54:12.162455 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0929 22:54:12.162534 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 22:54:12.162666 (+ 132us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 22:54:12.162666 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 22:54:12.162668 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 22:54:12.162732 (+ 64us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 22:54:12.162734 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 22:54:12.163329 (+ 595us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 22:54:12.163331 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 22:54:12.164181 (+ 850us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 22:54:12.164184 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 22:54:12.164718 (+ 534us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 22:54:12.164719 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 22:54:12.164793 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 22:54:12.164796 (+ 3us) client_negotiation.cc:769] Sending connection context
0929 22:54:12.164845 (+ 49us) client_negotiation.cc:240] Negotiation successful
0929 22:54:12.164894 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":174,"mutex_wait_us":4,"spinlock_wait_cycles":16128,"thread_start_us":80,"threads_started":1}
I20260929 22:54:12.165365 2571 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fe4fdcedc6094fb6afeaebef7c533bcb" candidate_uuid: "c8f0afee5cf34e398dcdd5ac051df6af" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f8e06c7affda4c13b29a9dc716c34d96" is_pre_election: true
I20260929 22:54:12.165535 2571 raft_consensus.cc:2454] T fe4fdcedc6094fb6afeaebef7c533bcb P f8e06c7affda4c13b29a9dc716c34d96 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c8f0afee5cf34e398dcdd5ac051df6af in term 0.
I20260929 22:54:12.165515 2309 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fe4fdcedc6094fb6afeaebef7c533bcb" candidate_uuid: "c8f0afee5cf34e398dcdd5ac051df6af" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1442caba435a4bcc8a7030a781505d75" is_pre_election: true
I20260929 22:54:12.165654 2309 raft_consensus.cc:2454] T fe4fdcedc6094fb6afeaebef7c533bcb P 1442caba435a4bcc8a7030a781505d75 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c8f0afee5cf34e398dcdd5ac051df6af in term 0.
I20260929 22:54:12.165751 2376 leader_election.cc:302] T fe4fdcedc6094fb6afeaebef7c533bcb P c8f0afee5cf34e398dcdd5ac051df6af [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c8f0afee5cf34e398dcdd5ac051df6af, f8e06c7affda4c13b29a9dc716c34d96; no voters:
I20260929 22:54:12.165925 2638 raft_consensus.cc:2781] T fe4fdcedc6094fb6afeaebef7c533bcb P c8f0afee5cf34e398dcdd5ac051df6af [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 22:54:12.165990 2638 raft_consensus.cc:482] T fe4fdcedc6094fb6afeaebef7c533bcb P c8f0afee5cf34e398dcdd5ac051df6af [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 22:54:12.166013 2638 raft_consensus.cc:3037] T fe4fdcedc6094fb6afeaebef7c533bcb P c8f0afee5cf34e398dcdd5ac051df6af [term 0 FOLLOWER]: Advancing to term 1
I20260929 22:54:12.166706 2638 raft_consensus.cc:504] T fe4fdcedc6094fb6afeaebef7c533bcb P c8f0afee5cf34e398dcdd5ac051df6af [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1442caba435a4bcc8a7030a781505d75" member_type: VOTER last_known_addr { host: "127.14.147.193" port: 40521 } } peers { permanent_uuid: "f8e06c7affda4c13b29a9dc716c34d96" member_type: VOTER last_known_addr { host: "127.14.147.195" port: 34193 } } peers { permanent_uuid: "c8f0afee5cf34e398dcdd5ac051df6af" member_type: VOTER last_known_addr { host: "127.14.147.194" port: 45741 } }
I20260929 22:54:12.166831 2638 leader_election.cc:288] T fe4fdcedc6094fb6afeaebef7c533bcb P c8f0afee5cf34e398dcdd5ac051df6af [CANDIDATE]: Term 1 election: Requested vote from peers 1442caba435a4bcc8a7030a781505d75 (127.14.147.193:40521), f8e06c7affda4c13b29a9dc716c34d96 (127.14.147.195:34193)
I20260929 22:54:12.166985 2309 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fe4fdcedc6094fb6afeaebef7c533bcb" candidate_uuid: "c8f0afee5cf34e398dcdd5ac051df6af" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1442caba435a4bcc8a7030a781505d75"
I20260929 22:54:12.167021 2571 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fe4fdcedc6094fb6afeaebef7c533bcb" candidate_uuid: "c8f0afee5cf34e398dcdd5ac051df6af" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f8e06c7affda4c13b29a9dc716c34d96"
I20260929 22:54:12.167070 2309 raft_consensus.cc:3037] T fe4fdcedc6094fb6afeaebef7c533bcb P 1442caba435a4bcc8a7030a781505d75 [term 0 FOLLOWER]: Advancing to term 1
I20260929 22:54:12.167083 2571 raft_consensus.cc:3037] T fe4fdcedc6094fb6afeaebef7c533bcb P f8e06c7affda4c13b29a9dc716c34d96 [term 0 FOLLOWER]: Advancing to term 1
I20260929 22:54:12.167985 2571 raft_consensus.cc:2454] T fe4fdcedc6094fb6afeaebef7c533bcb P f8e06c7affda4c13b29a9dc716c34d96 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c8f0afee5cf34e398dcdd5ac051df6af in term 1.
I20260929 22:54:12.167985 2309 raft_consensus.cc:2454] T fe4fdcedc6094fb6afeaebef7c533bcb P 1442caba435a4bcc8a7030a781505d75 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c8f0afee5cf34e398dcdd5ac051df6af in term 1.
I20260929 22:54:12.168143 2376 leader_election.cc:302] T fe4fdcedc6094fb6afeaebef7c533bcb P c8f0afee5cf34e398dcdd5ac051df6af [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c8f0afee5cf34e398dcdd5ac051df6af, f8e06c7affda4c13b29a9dc716c34d96; no voters:
I20260929 22:54:12.168219 2638 raft_consensus.cc:2781] T fe4fdcedc6094fb6afeaebef7c533bcb P c8f0afee5cf34e398dcdd5ac051df6af [term 1 FOLLOWER]: Leader election won for term 1
I20260929 22:54:12.168354 2638 raft_consensus.cc:686] T fe4fdcedc6094fb6afeaebef7c533bcb P c8f0afee5cf34e398dcdd5ac051df6af [term 1 LEADER]: Becoming Leader. State: Replica: c8f0afee5cf34e398dcdd5ac051df6af, State: Running, Role: LEADER
I20260929 22:54:12.168437 2638 consensus_queue.cc:237] T fe4fdcedc6094fb6afeaebef7c533bcb P c8f0afee5cf34e398dcdd5ac051df6af [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1442caba435a4bcc8a7030a781505d75" member_type: VOTER last_known_addr { host: "127.14.147.193" port: 40521 } } peers { permanent_uuid: "f8e06c7affda4c13b29a9dc716c34d96" member_type: VOTER last_known_addr { host: "127.14.147.195" port: 34193 } } peers { permanent_uuid: "c8f0afee5cf34e398dcdd5ac051df6af" member_type: VOTER last_known_addr { host: "127.14.147.194" port: 45741 } }
I20260929 22:54:12.169253 2166 catalog_manager.cc:5495] T fe4fdcedc6094fb6afeaebef7c533bcb P c8f0afee5cf34e398dcdd5ac051df6af reported cstate change: term changed from 0 to 1, leader changed from <none> to c8f0afee5cf34e398dcdd5ac051df6af (127.14.147.194). New cstate: current_term: 1 leader_uuid: "c8f0afee5cf34e398dcdd5ac051df6af" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1442caba435a4bcc8a7030a781505d75" member_type: VOTER last_known_addr { host: "127.14.147.193" port: 40521 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f8e06c7affda4c13b29a9dc716c34d96" member_type: VOTER last_known_addr { host: "127.14.147.195" port: 34193 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c8f0afee5cf34e398dcdd5ac051df6af" member_type: VOTER last_known_addr { host: "127.14.147.194" port: 45741 } health_report { overall_health: HEALTHY } } }
I20260929 22:54:12.198769 2629 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:54:12.196348 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53832 (local address 127.14.147.194:45741)
0929 22:54:12.196411 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0929 22:54:12.196413 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 22:54:12.196421 (+ 8us) server_negotiation.cc:407] Connection header received
0929 22:54:12.196463 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 22:54:12.196464 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 22:54:12.196482 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 22:54:12.196556 (+ 74us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 22:54:12.196755 (+ 199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:54:12.197231 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:54:12.198014 (+ 783us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:54:12.198216 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:54:12.198277 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 22:54:12.198392 (+ 115us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 22:54:12.198496 (+ 104us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 22:54:12.198552 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0929 22:54:12.198689 (+ 137us) server_negotiation.cc:299] Negotiation successful
0929 22:54:12.198705 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260929 22:54:12.200721 2571 raft_consensus.cc:1261] T fe4fdcedc6094fb6afeaebef7c533bcb P f8e06c7affda4c13b29a9dc716c34d96 [term 1 FOLLOWER]: Refusing update from remote peer c8f0afee5cf34e398dcdd5ac051df6af: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 22:54:12.200724 2309 raft_consensus.cc:1261] T fe4fdcedc6094fb6afeaebef7c533bcb P 1442caba435a4bcc8a7030a781505d75 [term 1 FOLLOWER]: Refusing update from remote peer c8f0afee5cf34e398dcdd5ac051df6af: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 22:54:12.201076 2642 consensus_queue.cc:1035] T fe4fdcedc6094fb6afeaebef7c533bcb P c8f0afee5cf34e398dcdd5ac051df6af [LEADER]: Connected to new peer: Peer: permanent_uuid: "1442caba435a4bcc8a7030a781505d75" member_type: VOTER last_known_addr { host: "127.14.147.193" port: 40521 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 22:54:12.201217 2638 consensus_queue.cc:1035] T fe4fdcedc6094fb6afeaebef7c533bcb P c8f0afee5cf34e398dcdd5ac051df6af [LEADER]: Connected to new peer: Peer: permanent_uuid: "f8e06c7affda4c13b29a9dc716c34d96" member_type: VOTER last_known_addr { host: "127.14.147.195" port: 34193 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 22:54:12.203840 2645 mvcc.cc:205] Tried to move back new op lower bound from 7334799164211200000 to 7334799164082532352. Current Snapshot: MvccSnapshot[applied={T|T < 7334799164211200000}]
I20260929 22:54:12.207295 2167 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:59274:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 22:54:12.207396 2167 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:59274:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 22:54:12.208181 2167 catalog_manager.cc:5782] T 00000000000000000000000000000000 P c5174aff7fe34574aa07ea9e5dd38d69: Sending DeleteTablet for 3 replicas of tablet fe4fdcedc6094fb6afeaebef7c533bcb
I20260929 22:54:12.208608 2551 tablet_service.cc:1514] Processing DeleteTablet for tablet fe4fdcedc6094fb6afeaebef7c533bcb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 22:54:12 UTC) from {username='slave'} at 127.0.0.1:58146
I20260929 22:54:12.208595 2289 tablet_service.cc:1514] Processing DeleteTablet for tablet fe4fdcedc6094fb6afeaebef7c533bcb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 22:54:12 UTC) from {username='slave'} at 127.0.0.1:35648
I20260929 22:54:12.208843 2420 tablet_service.cc:1514] Processing DeleteTablet for tablet fe4fdcedc6094fb6afeaebef7c533bcb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 22:54:12 UTC) from {username='slave'} at 127.0.0.1:53822
I20260929 22:54:12.209066 2657 tablet_replica.cc:330] T fe4fdcedc6094fb6afeaebef7c533bcb P c8f0afee5cf34e398dcdd5ac051df6af: stopping tablet replica
I20260929 22:54:12.209223 2657 raft_consensus.cc:2229] T fe4fdcedc6094fb6afeaebef7c533bcb P c8f0afee5cf34e398dcdd5ac051df6af [term 1 LEADER]: Raft consensus shutting down.
I20260929 22:54:12.209268 14927 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhWOCLk/build/release/bin/kudu with pid 2227
I20260929 22:54:12.209436 2657 raft_consensus.cc:2258] T fe4fdcedc6094fb6afeaebef7c533bcb P c8f0afee5cf34e398dcdd5ac051df6af [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 22:54:12.210739 2657 ts_tablet_manager.cc:1905] T fe4fdcedc6094fb6afeaebef7c533bcb P c8f0afee5cf34e398dcdd5ac051df6af: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 22:54:12.212374 2656 tablet_replica.cc:330] T fe4fdcedc6094fb6afeaebef7c533bcb P f8e06c7affda4c13b29a9dc716c34d96: stopping tablet replica
I20260929 22:54:12.212504 2656 raft_consensus.cc:2229] T fe4fdcedc6094fb6afeaebef7c533bcb P f8e06c7affda4c13b29a9dc716c34d96 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 22:54:12.212589 2657 ts_tablet_manager.cc:1918] T fe4fdcedc6094fb6afeaebef7c533bcb P c8f0afee5cf34e398dcdd5ac051df6af: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 22:54:12.212630 2656 raft_consensus.cc:2258] T fe4fdcedc6094fb6afeaebef7c533bcb P f8e06c7affda4c13b29a9dc716c34d96 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 22:54:12.212671 2657 log.cc:1196] T fe4fdcedc6094fb6afeaebef7c533bcb P c8f0afee5cf34e398dcdd5ac051df6af: Deleting WAL directory at /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790722213368738-14927-0/minicluster-data/ts-1/wal/wals/fe4fdcedc6094fb6afeaebef7c533bcb
I20260929 22:54:12.212965 2657 ts_tablet_manager.cc:1939] T fe4fdcedc6094fb6afeaebef7c533bcb P c8f0afee5cf34e398dcdd5ac051df6af: Deleting consensus metadata
I20260929 22:54:12.213145 2656 ts_tablet_manager.cc:1905] T fe4fdcedc6094fb6afeaebef7c533bcb P f8e06c7affda4c13b29a9dc716c34d96: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 22:54:12.213609 2152 catalog_manager.cc:4841] TS c8f0afee5cf34e398dcdd5ac051df6af (127.14.147.194:45741): tablet fe4fdcedc6094fb6afeaebef7c533bcb (table test-table [id=a4abccf85ef24c7eaab888f5f017a7ee]) successfully deleted
I20260929 22:54:12.214759 2656 ts_tablet_manager.cc:1918] T fe4fdcedc6094fb6afeaebef7c533bcb P f8e06c7affda4c13b29a9dc716c34d96: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 22:54:12.214843 2656 log.cc:1196] T fe4fdcedc6094fb6afeaebef7c533bcb P f8e06c7affda4c13b29a9dc716c34d96: Deleting WAL directory at /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790722213368738-14927-0/minicluster-data/ts-2/wal/wals/fe4fdcedc6094fb6afeaebef7c533bcb
I20260929 22:54:12.215065 2656 ts_tablet_manager.cc:1939] T fe4fdcedc6094fb6afeaebef7c533bcb P f8e06c7affda4c13b29a9dc716c34d96: Deleting consensus metadata
I20260929 22:54:12.215731 2154 catalog_manager.cc:4841] TS f8e06c7affda4c13b29a9dc716c34d96 (127.14.147.195:34193): tablet fe4fdcedc6094fb6afeaebef7c533bcb (table test-table [id=a4abccf85ef24c7eaab888f5f017a7ee]) successfully deleted
W20260929 22:54:12.216475 2153 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.14.147.193:40521 (error 108)
I20260929 22:54:12.216641 14927 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhWOCLk/build/release/bin/kudu with pid 2358
I20260929 22:54:12.218123 2626 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:54:12.217986 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.147.193:40521 (local address 127.0.0.1:35662)
0929 22:54:12.218029 (+ 43us) negotiation.cc:107] Waiting for socket to connect
0929 22:54:12.218083 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.147.193:40521: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":19}
W20260929 22:54:12.218250 2153 catalog_manager.cc:4584] TS 1442caba435a4bcc8a7030a781505d75 (127.14.147.193:40521): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet fe4fdcedc6094fb6afeaebef7c533bcb: Network error: Client connection negotiation failed: client connection to 127.14.147.193:40521: connect: Connection refused (error 111)
I20260929 22:54:12.220839 14927 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhWOCLk/build/release/bin/kudu with pid 2489
I20260929 22:54:12.225409 14927 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhWOCLk/build/release/bin/kudu with pid 2136
2026-09-29T22:54:12Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/5 (1026 ms)
[----------] 6 tests from EncryptionPolicyTest (8064 ms total)
[----------] 18 tests from AuthTokenIssuingTest
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/0
Loading random data
Initializing database '/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790722213368738-14927-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 22:54:12 dist-test-slave-98mk krb5kdc[2661](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 22:54:12 dist-test-slave-98mk krb5kdc[2661](info): set up 2 sockets
Sep 29 22:54:12 dist-test-slave-98mk krb5kdc[2661](info): commencing operation
krb5kdc: starting...
W20260929 22:54:13.167981 14927 mini_kdc.cc:121] Time spent starting KDC: real 0.924s user 0.001s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790722213368738-14927-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790722213368738-14927-0/krb5kdc/test-user.keytab.
Sep 29 22:54:13 dist-test-slave-98mk krb5kdc[2661](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790722453, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T22:54:13Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T22:54:13Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790722213368738-14927-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790722213368738-14927-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260929 22:54:13.279668 14927 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhWOCLk/build/release/bin/kudu
/tmp/dist-test-taskhWOCLk/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790722213368738-14927-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790722213368738-14927-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790722213368738-14927-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790722213368738-14927-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46337
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790722213368738-14927-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:35019
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:46337
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790722213368738-14927-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790722213368738-14927-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790722213368738-14927-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 22:54:13.359934 2674 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 22:54:13.360162 2674 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 22:54:13.360194 2674 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 22:54:13.361402 2674 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 22:54:13.361450 2674 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 22:54:13.361469 2674 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 22:54:13.361485 2674 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 22:54:13.361500 2674 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 22:54:13.363029 2674 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:35019
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790722213368738-14927-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790722213368738-14927-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:46337
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790722213368738-14927-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46337
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790722213368738-14927-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2674
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790722213368738-14927-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 21:58:54 UTC on ccycloud.quasar-xguoti.root.comops.site
build id 4681390
I20260929 22:54:13.363284 2674 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 22:54:13.363536 2674 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 22:54:13.368161 2679 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 22:54:13.368156 2682 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 22:54:13.368156 2680 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 22:54:13.368371 2674 server_base.cc:1034] running on GCE node
I20260929 22:54:13.368611 2674 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 22:54:13.368897 2674 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 22:54:13.370046 2674 hybrid_clock.cc:648] HybridClock initialized: now 1790722453370019 us; error 62 us; skew 500 ppm
Sep 29 22:54:13 dist-test-slave-98mk krb5kdc[2661](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790722453, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 22:54:13.372783 2674 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260929 22:54:13.373250 2674 webserver.cc:460] Webserver started at http://127.0.0.1:33111/ using document root <none> and password file <none>
I20260929 22:54:13.373457 2674 fs_manager.cc:359] Metadata directory not provided
I20260929 22:54:13.373502 2674 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 22:54:13.373631 2674 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 22:54:13.374820 2674 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790722213368738-14927-0/minicluster-data/master-0/data/instance:
uuid: "fbdef6d036c042498659c906ee8dd401"
format_stamp: "Formatted at 2026-09-29 22:54:13 on dist-test-slave-98mk"
server_key: "38cc9ff169ca879ae6630251d821b19f"
server_key_iv: "470bbbb0b158aac3b1012f2135b51415"
server_key_version: "encryptionkey@0"
I20260929 22:54:13.375169 2674 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790722213368738-14927-0/minicluster-data/master-0/wal/instance:
uuid: "fbdef6d036c042498659c906ee8dd401"
format_stamp: "Formatted at 2026-09-29 22:54:13 on dist-test-slave-98mk"
server_key: "38cc9ff169ca879ae6630251d821b19f"
server_key_iv: "470bbbb0b158aac3b1012f2135b51415"
server_key_version: "encryptionkey@0"
I20260929 22:54:13.376624 2674 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260929 22:54:13.377543 2689 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 22:54:13.377804 2674 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260929 22:54:13.377900 2674 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790722213368738-14927-0/minicluster-data/master-0/data,/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790722213368738-14927-0/minicluster-data/master-0/wal
uuid: "fbdef6d036c042498659c906ee8dd401"
format_stamp: "Formatted at 2026-09-29 22:54:13 on dist-test-slave-98mk"
server_key: "38cc9ff169ca879ae6630251d821b19f"
server_key_iv: "470bbbb0b158aac3b1012f2135b51415"
server_key_version: "encryptionkey@0"
I20260929 22:54:13.377981 2674 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790722213368738-14927-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790722213368738-14927-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790722213368738-14927-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 22:54:13.387941 2674 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 22:54:13.389739 2674 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 22:54:13.389864 2674 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 22:54:13.393731 2674 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:46337
I20260929 22:54:13.393769 2741 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:46337 every 8 connection(s)
I20260929 22:54:13.394098 2674 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790722213368738-14927-0/minicluster-data/master-0/data/info.pb
I20260929 22:54:13.394407 14927 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhWOCLk/build/release/bin/kudu as pid 2674
I20260929 22:54:13.394523 14927 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790722213368738-14927-0/minicluster-data/master-0/wal/instance
I20260929 22:54:13.394714 14927 external_mini_cluster.cc:1428] Setting key 12e6b5db43e0adb0cc49287bf20b9bb5
I20260929 22:54:13.395004 2742 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 22:54:13.397861 2742 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fbdef6d036c042498659c906ee8dd401: Bootstrap starting.
I20260929 22:54:13.398481 2742 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P fbdef6d036c042498659c906ee8dd401: Neither blocks nor log segments found. Creating new log.
I20260929 22:54:13.398794 2742 log.cc:824] T 00000000000000000000000000000000 P fbdef6d036c042498659c906ee8dd401: Log is configured to *not* fsync() on all Append() calls
I20260929 22:54:13.399961 2742 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fbdef6d036c042498659c906ee8dd401: No bootstrap required, opened a new log
Sep 29 22:54:13 dist-test-slave-98mk krb5kdc[2661](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790722453, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260929 22:54:13.401872 2742 raft_consensus.cc:348] T 00000000000000000000000000000000 P fbdef6d036c042498659c906ee8dd401 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fbdef6d036c042498659c906ee8dd401" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46337 } }
I20260929 22:54:13.402041 2742 raft_consensus.cc:374] T 00000000000000000000000000000000 P fbdef6d036c042498659c906ee8dd401 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 22:54:13.402076 2742 raft_consensus.cc:729] T 00000000000000000000000000000000 P fbdef6d036c042498659c906ee8dd401 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fbdef6d036c042498659c906ee8dd401, State: Initialized, Role: FOLLOWER
I20260929 22:54:13.402208 2742 consensus_queue.cc:260] T 00000000000000000000000000000000 P fbdef6d036c042498659c906ee8dd401 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fbdef6d036c042498659c906ee8dd401" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46337 } }
I20260929 22:54:13.402283 2742 raft_consensus.cc:388] T 00000000000000000000000000000000 P fbdef6d036c042498659c906ee8dd401 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 22:54:13.402343 2742 raft_consensus.cc:482] T 00000000000000000000000000000000 P fbdef6d036c042498659c906ee8dd401 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 22:54:13.402397 2742 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fbdef6d036c042498659c906ee8dd401 [term 0 FOLLOWER]: Advancing to term 1
I20260929 22:54:13.403014 2742 raft_consensus.cc:504] T 00000000000000000000000000000000 P fbdef6d036c042498659c906ee8dd401 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fbdef6d036c042498659c906ee8dd401" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46337 } }
I20260929 22:54:13.403110 2742 leader_election.cc:302] T 00000000000000000000000000000000 P fbdef6d036c042498659c906ee8dd401 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: fbdef6d036c042498659c906ee8dd401; no voters:
I20260929 22:54:13.403296 2742 leader_election.cc:288] T 00000000000000000000000000000000 P fbdef6d036c042498659c906ee8dd401 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 22:54:13.403342 2747 raft_consensus.cc:2781] T 00000000000000000000000000000000 P fbdef6d036c042498659c906ee8dd401 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 22:54:13.403491 2747 raft_consensus.cc:686] T 00000000000000000000000000000000 P fbdef6d036c042498659c906ee8dd401 [term 1 LEADER]: Becoming Leader. State: Replica: fbdef6d036c042498659c906ee8dd401, State: Running, Role: LEADER
I20260929 22:54:13.403594 2742 sys_catalog.cc:564] T 00000000000000000000000000000000 P fbdef6d036c042498659c906ee8dd401 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 22:54:13.403646 2747 consensus_queue.cc:237] T 00000000000000000000000000000000 P fbdef6d036c042498659c906ee8dd401 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fbdef6d036c042498659c906ee8dd401" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46337 } }
I20260929 22:54:13.404095 2749 sys_catalog.cc:455] T 00000000000000000000000000000000 P fbdef6d036c042498659c906ee8dd401 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "fbdef6d036c042498659c906ee8dd401" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fbdef6d036c042498659c906ee8dd401" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46337 } } }
I20260929 22:54:13.404181 2749 sys_catalog.cc:458] T 00000000000000000000000000000000 P fbdef6d036c042498659c906ee8dd401 [sys.catalog]: This master's current role is: LEADER
I20260929 22:54:13.404402 2747 sys_catalog.cc:455] T 00000000000000000000000000000000 P fbdef6d036c042498659c906ee8dd401 [sys.catalog]: SysCatalogTable state changed. Reason: New leader fbdef6d036c042498659c906ee8dd401. Latest consensus state: current_term: 1 leader_uuid: "fbdef6d036c042498659c906ee8dd401" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fbdef6d036c042498659c906ee8dd401" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46337 } } }
I20260929 22:54:13.404520 2747 sys_catalog.cc:458] T 00000000000000000000000000000000 P fbdef6d036c042498659c906ee8dd401 [sys.catalog]: This master's current role is: LEADER
I20260929 22:54:13.404538 2756 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 22:54:13.405138 2756 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 22:54:13.406102 2745 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:54:13.395203 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46037 (local address 127.0.0.1:46337)
0929 22:54:13.395379 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0929 22:54:13.395393 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0929 22:54:13.395447 (+ 54us) server_negotiation.cc:407] Connection header received
0929 22:54:13.396146 (+ 699us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 22:54:13.396154 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 22:54:13.396213 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 22:54:13.396340 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0929 22:54:13.396945 (+ 605us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:54:13.397812 (+ 867us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:54:13.398582 (+ 770us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:54:13.399178 (+ 596us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:54:13.399278 (+ 100us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 22:54:13.402389 (+ 3111us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 22:54:13.402394 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 22:54:13.402394 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 22:54:13.402409 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0929 22:54:13.404323 (+ 1914us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 22:54:13.404836 (+ 513us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 22:54:13.404839 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 22:54:13.404839 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 22:54:13.404955 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 22:54:13.405218 (+ 263us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 22:54:13.405219 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 22:54:13.405219 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 22:54:13.405536 (+ 317us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 22:54:13.405697 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0929 22:54:13.405824 (+ 127us) server_negotiation.cc:299] Negotiation successful
0929 22:54:13.405932 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":131,"thread_start_us":86,"threads_started":1}
I20260929 22:54:13.406865 2756 catalog_manager.cc:1347] Generated new cluster ID: aafcda7f576c4a198bde878100d021ec
I20260929 22:54:13.406966 2756 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 22:54:13.407115 14927 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 22:54:13.416141 2756 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 22:54:13.416944 2756 catalog_manager.cc:1504] Loading token signing keys...
I20260929 22:54:13.417969 2745 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:54:13.411752 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53863 (local address 127.0.0.1:46337)
0929 22:54:13.411800 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0929 22:54:13.411802 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 22:54:13.411961 (+ 159us) server_negotiation.cc:407] Connection header received
0929 22:54:13.412010 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 22:54:13.412011 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 22:54:13.412021 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 22:54:13.412074 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0929 22:54:13.412615 (+ 541us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:54:13.413116 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:54:13.413797 (+ 681us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:54:13.414072 (+ 275us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:54:13.414164 (+ 92us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 22:54:13.415354 (+ 1190us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 22:54:13.415356 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 22:54:13.415356 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 22:54:13.415364 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0929 22:54:13.416861 (+ 1497us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 22:54:13.417242 (+ 381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 22:54:13.417243 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 22:54:13.417244 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 22:54:13.417301 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 22:54:13.417495 (+ 194us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 22:54:13.417496 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 22:54:13.417496 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 22:54:13.417604 (+ 108us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 22:54:13.417696 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0929 22:54:13.417809 (+ 113us) server_negotiation.cc:299] Negotiation successful
0929 22:54:13.417873 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260929 22:54:13.421608 2756 catalog_manager.cc:5868] T 00000000000000000000000000000000 P fbdef6d036c042498659c906ee8dd401: Generated new TSK 0
I20260929 22:54:13.421813 2756 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 22:54:13.425771 14927 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhWOCLk/build/release/bin/kudu with pid 2674
W20260929 22:54:13.429942 2659 connection.cc:535] client connection to 127.0.0.1:46337 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:46337): Connection reset by peer (error 104)
2026-09-29T22:54:13Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/0 (1199 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/1
Loading random data
Initializing database '/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790722213368738-14927-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 22:54:13 dist-test-slave-98mk krb5kdc[2773](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 22:54:13 dist-test-slave-98mk krb5kdc[2773](info): set up 2 sockets
Sep 29 22:54:13 dist-test-slave-98mk krb5kdc[2773](info): commencing operation
krb5kdc: starting...
W20260929 22:54:14.368049 14927 mini_kdc.cc:121] Time spent starting KDC: real 0.924s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790722213368738-14927-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790722213368738-14927-0/krb5kdc/test-user.keytab.
Sep 29 22:54:14 dist-test-slave-98mk krb5kdc[2773](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790722454, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T22:54:14Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T22:54:14Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790722213368738-14927-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790722213368738-14927-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260929 22:54:14.480261 14927 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhWOCLk/build/release/bin/kudu
/tmp/dist-test-taskhWOCLk/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790722213368738-14927-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790722213368738-14927-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790722213368738-14927-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790722213368738-14927-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38669
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790722213368738-14927-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:33155
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:38669
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790722213368738-14927-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790722213368738-14927-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790722213368738-14927-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 22:54:14.561028 2786 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 22:54:14.561264 2786 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 22:54:14.561295 2786 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 22:54:14.562510 2786 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 22:54:14.562561 2786 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 22:54:14.562578 2786 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 22:54:14.562595 2786 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 22:54:14.562609 2786 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 22:54:14.564190 2786 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:33155
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790722213368738-14927-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790722213368738-14927-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:38669
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790722213368738-14927-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38669
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790722213368738-14927-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2786
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790722213368738-14927-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 21:58:54 UTC on ccycloud.quasar-xguoti.root.comops.site
build id 4681390
I20260929 22:54:14.564459 2786 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 22:54:14.564671 2786 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 22:54:14.569368 2794 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 22:54:14.569379 2792 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 22:54:14.569362 2791 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 22:54:14.569742 2786 server_base.cc:1034] running on GCE node
I20260929 22:54:14.570070 2786 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 22:54:14.570377 2786 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 22:54:14.571530 2786 hybrid_clock.cc:648] HybridClock initialized: now 1790722454571512 us; error 28 us; skew 500 ppm
Sep 29 22:54:14 dist-test-slave-98mk krb5kdc[2773](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790722454, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 22:54:14.580132 2786 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260929 22:54:14.580651 2786 webserver.cc:460] Webserver started at http://127.0.0.1:33757/ using document root <none> and password file <none>
I20260929 22:54:14.580862 2786 fs_manager.cc:359] Metadata directory not provided
I20260929 22:54:14.580914 2786 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 22:54:14.581046 2786 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 22:54:14.582132 2786 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790722213368738-14927-0/minicluster-data/master-0/data/instance:
uuid: "b4a674d74d004ee080c441fc054e82c8"
format_stamp: "Formatted at 2026-09-29 22:54:14 on dist-test-slave-98mk"
server_key: "f34e6e415355bfd993b7dad3f295dc28"
server_key_iv: "50e5d573cc99a496ea5c0c30dc1bcf8a"
server_key_version: "encryptionkey@0"
I20260929 22:54:14.582484 2786 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790722213368738-14927-0/minicluster-data/master-0/wal/instance:
uuid: "b4a674d74d004ee080c441fc054e82c8"
format_stamp: "Formatted at 2026-09-29 22:54:14 on dist-test-slave-98mk"
server_key: "f34e6e415355bfd993b7dad3f295dc28"
server_key_iv: "50e5d573cc99a496ea5c0c30dc1bcf8a"
server_key_version: "encryptionkey@0"
I20260929 22:54:14.583832 2786 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260929 22:54:14.584537 2801 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 22:54:14.584688 2786 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260929 22:54:14.584774 2786 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790722213368738-14927-0/minicluster-data/master-0/data,/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790722213368738-14927-0/minicluster-data/master-0/wal
uuid: "b4a674d74d004ee080c441fc054e82c8"
format_stamp: "Formatted at 2026-09-29 22:54:14 on dist-test-slave-98mk"
server_key: "f34e6e415355bfd993b7dad3f295dc28"
server_key_iv: "50e5d573cc99a496ea5c0c30dc1bcf8a"
server_key_version: "encryptionkey@0"
I20260929 22:54:14.584834 2786 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790722213368738-14927-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790722213368738-14927-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790722213368738-14927-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 22:54:14.608326 2786 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 22:54:14.610240 2786 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 22:54:14.610376 2786 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 22:54:14.614382 2786 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:38669
I20260929 22:54:14.614426 2853 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:38669 every 8 connection(s)
I20260929 22:54:14.614786 2786 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790722213368738-14927-0/minicluster-data/master-0/data/info.pb
I20260929 22:54:14.615568 14927 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhWOCLk/build/release/bin/kudu as pid 2786
I20260929 22:54:14.615686 14927 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790722213368738-14927-0/minicluster-data/master-0/wal/instance
I20260929 22:54:14.615808 2854 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 22:54:14.615883 14927 external_mini_cluster.cc:1428] Setting key d964446b797f95f3b99df0f9d8bff602
I20260929 22:54:14.618985 2854 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b4a674d74d004ee080c441fc054e82c8: Bootstrap starting.
I20260929 22:54:14.619716 2854 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b4a674d74d004ee080c441fc054e82c8: Neither blocks nor log segments found. Creating new log.
I20260929 22:54:14.619977 2854 log.cc:824] T 00000000000000000000000000000000 P b4a674d74d004ee080c441fc054e82c8: Log is configured to *not* fsync() on all Append() calls
I20260929 22:54:14.620667 2854 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b4a674d74d004ee080c441fc054e82c8: No bootstrap required, opened a new log
I20260929 22:54:14.621490 2857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:54:14.616386 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42853 (local address 127.0.0.1:38669)
0929 22:54:14.616557 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0929 22:54:14.616565 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 22:54:14.616589 (+ 24us) server_negotiation.cc:407] Connection header received
0929 22:54:14.617205 (+ 616us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 22:54:14.617213 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 22:54:14.617261 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 22:54:14.617377 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0929 22:54:14.617630 (+ 253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:54:14.618497 (+ 867us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:54:14.619565 (+ 1068us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:54:14.619998 (+ 433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:54:14.620106 (+ 108us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 22:54:14.620781 (+ 675us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 22:54:14.620786 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 22:54:14.620786 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 22:54:14.621003 (+ 217us) server_negotiation.cc:922] calling sasl_server_start()
0929 22:54:14.621042 (+ 39us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 22:54:14.621057 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 22:54:14.621142 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0929 22:54:14.621236 (+ 94us) server_negotiation.cc:299] Negotiation successful
0929 22:54:14.621265 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":121,"thread_start_us":73,"threads_started":1}
I20260929 22:54:14.622629 14927 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 22:54:14.622900 2854 raft_consensus.cc:348] T 00000000000000000000000000000000 P b4a674d74d004ee080c441fc054e82c8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b4a674d74d004ee080c441fc054e82c8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38669 } }
I20260929 22:54:14.623040 2854 raft_consensus.cc:374] T 00000000000000000000000000000000 P b4a674d74d004ee080c441fc054e82c8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 22:54:14.623063 2854 raft_consensus.cc:729] T 00000000000000000000000000000000 P b4a674d74d004ee080c441fc054e82c8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b4a674d74d004ee080c441fc054e82c8, State: Initialized, Role: FOLLOWER
I20260929 22:54:14.623178 2854 consensus_queue.cc:260] T 00000000000000000000000000000000 P b4a674d74d004ee080c441fc054e82c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b4a674d74d004ee080c441fc054e82c8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38669 } }
I20260929 22:54:14.623239 2854 raft_consensus.cc:388] T 00000000000000000000000000000000 P b4a674d74d004ee080c441fc054e82c8 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 22:54:14.623268 2854 raft_consensus.cc:482] T 00000000000000000000000000000000 P b4a674d74d004ee080c441fc054e82c8 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 22:54:14.623314 2854 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b4a674d74d004ee080c441fc054e82c8 [term 0 FOLLOWER]: Advancing to term 1
I20260929 22:54:14.623972 2854 raft_consensus.cc:504] T 00000000000000000000000000000000 P b4a674d74d004ee080c441fc054e82c8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b4a674d74d004ee080c441fc054e82c8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38669 } }
I20260929 22:54:14.624080 2854 leader_election.cc:302] T 00000000000000000000000000000000 P b4a674d74d004ee080c441fc054e82c8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b4a674d74d004ee080c441fc054e82c8; no voters:
I20260929 22:54:14.624245 2854 leader_election.cc:288] T 00000000000000000000000000000000 P b4a674d74d004ee080c441fc054e82c8 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 22:54:14.624413 2865 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b4a674d74d004ee080c441fc054e82c8 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 22:54:14.624511 2854 sys_catalog.cc:564] T 00000000000000000000000000000000 P b4a674d74d004ee080c441fc054e82c8 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 22:54:14.624584 2865 raft_consensus.cc:686] T 00000000000000000000000000000000 P b4a674d74d004ee080c441fc054e82c8 [term 1 LEADER]: Becoming Leader. State: Replica: b4a674d74d004ee080c441fc054e82c8, State: Running, Role: LEADER
I20260929 22:54:14.624720 2865 consensus_queue.cc:237] T 00000000000000000000000000000000 P b4a674d74d004ee080c441fc054e82c8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b4a674d74d004ee080c441fc054e82c8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38669 } }
I20260929 22:54:14.625118 2864 sys_catalog.cc:455] T 00000000000000000000000000000000 P b4a674d74d004ee080c441fc054e82c8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b4a674d74d004ee080c441fc054e82c8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b4a674d74d004ee080c441fc054e82c8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38669 } } }
I20260929 22:54:14.625185 2864 sys_catalog.cc:458] T 00000000000000000000000000000000 P b4a674d74d004ee080c441fc054e82c8 [sys.catalog]: This master's current role is: LEADER
I20260929 22:54:14.625327 2866 sys_catalog.cc:455] T 00000000000000000000000000000000 P b4a674d74d004ee080c441fc054e82c8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b4a674d74d004ee080c441fc054e82c8. Latest consensus state: current_term: 1 leader_uuid: "b4a674d74d004ee080c441fc054e82c8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b4a674d74d004ee080c441fc054e82c8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38669 } } }
I20260929 22:54:14.625379 2866 sys_catalog.cc:458] T 00000000000000000000000000000000 P b4a674d74d004ee080c441fc054e82c8 [sys.catalog]: This master's current role is: LEADER
I20260929 22:54:14.625563 2873 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 22:54:14.626155 2873 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 22:54:14.627470 2857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:54:14.624055 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57451 (local address 127.0.0.1:38669)
0929 22:54:14.624096 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0929 22:54:14.624098 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 22:54:14.624182 (+ 84us) server_negotiation.cc:407] Connection header received
0929 22:54:14.624267 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 22:54:14.624268 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 22:54:14.624279 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 22:54:14.624347 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0929 22:54:14.624514 (+ 167us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:54:14.625196 (+ 682us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:54:14.626174 (+ 978us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:54:14.626452 (+ 278us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:54:14.626527 (+ 75us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 22:54:14.627206 (+ 679us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 22:54:14.627207 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 22:54:14.627207 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 22:54:14.627213 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0929 22:54:14.627229 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 22:54:14.627233 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 22:54:14.627294 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0929 22:54:14.627372 (+ 78us) server_negotiation.cc:299] Negotiation successful
0929 22:54:14.627383 (+ 11us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260929 22:54:14.627836 2873 catalog_manager.cc:1347] Generated new cluster ID: ec039c8f9e474c969c8f6b10ba3da0ef
I20260929 22:54:14.627887 2873 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 22:54:14.632180 2873 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 22:54:14.632678 2873 catalog_manager.cc:1504] Loading token signing keys...
I20260929 22:54:14.636642 2873 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b4a674d74d004ee080c441fc054e82c8: Generated new TSK 0
I20260929 22:54:14.636811 2873 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 22:54:14.638645 14927 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhWOCLk/build/release/bin/kudu with pid 2786
W20260929 22:54:14.642239 2771 connection.cc:535] client connection to 127.0.0.1:38669 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:38669): Connection reset by peer (error 104)
2026-09-29T22:54:14Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/1 (1211 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/2
Loading random data
Initializing database '/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790722213368738-14927-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 22:54:14 dist-test-slave-98mk krb5kdc[2885](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 22:54:14 dist-test-slave-98mk krb5kdc[2885](info): set up 2 sockets
Sep 29 22:54:14 dist-test-slave-98mk krb5kdc[2885](info): commencing operation
krb5kdc: starting...
W20260929 22:54:15.583900 14927 mini_kdc.cc:121] Time spent starting KDC: real 0.928s user 0.003s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790722213368738-14927-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790722213368738-14927-0/krb5kdc/test-user.keytab.
Sep 29 22:54:15 dist-test-slave-98mk krb5kdc[2885](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790722455, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T22:54:15Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T22:54:15Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790722213368738-14927-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790722213368738-14927-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260929 22:54:15.700502 14927 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhWOCLk/build/release/bin/kudu
/tmp/dist-test-taskhWOCLk/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790722213368738-14927-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790722213368738-14927-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790722213368738-14927-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790722213368738-14927-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39345
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790722213368738-14927-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:33751
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:39345
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790722213368738-14927-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790722213368738-14927-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790722213368738-14927-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 22:54:15.784238 2898 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 22:54:15.784472 2898 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 22:54:15.784497 2898 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 22:54:15.785692 2898 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 22:54:15.785741 2898 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 22:54:15.785755 2898 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 22:54:15.785768 2898 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 22:54:15.785790 2898 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 22:54:15.787315 2898 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:33751
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790722213368738-14927-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790722213368738-14927-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:39345
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790722213368738-14927-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39345
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790722213368738-14927-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2898
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790722213368738-14927-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 21:58:54 UTC on ccycloud.quasar-xguoti.root.comops.site
build id 4681390
I20260929 22:54:15.787582 2898 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 22:54:15.787793 2898 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 22:54:15.792383 2904 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 22:54:15.792392 2903 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 22:54:15.792426 2898 server_base.cc:1034] running on GCE node
W20260929 22:54:15.792418 2906 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 22:54:15.792829 2898 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 22:54:15.793105 2898 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 22:54:15.794245 2898 hybrid_clock.cc:648] HybridClock initialized: now 1790722455794232 us; error 29 us; skew 500 ppm
Sep 29 22:54:15 dist-test-slave-98mk krb5kdc[2885](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790722455, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 22:54:15.796787 2898 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260929 22:54:15.797171 2898 webserver.cc:460] Webserver started at http://127.0.0.1:34443/ using document root <none> and password file <none>
I20260929 22:54:15.797343 2898 fs_manager.cc:359] Metadata directory not provided
I20260929 22:54:15.797379 2898 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 22:54:15.797473 2898 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 22:54:15.798477 2898 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790722213368738-14927-0/minicluster-data/master-0/data/instance:
uuid: "ac05b5a84b9e431fa3c28dafcf775c5c"
format_stamp: "Formatted at 2026-09-29 22:54:15 on dist-test-slave-98mk"
server_key: "875d986e3f8f6a90e2dbf3bee417ebf9"
server_key_iv: "e74848311bfb8fd5b4591c3168c861de"
server_key_version: "encryptionkey@0"
I20260929 22:54:15.798763 2898 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790722213368738-14927-0/minicluster-data/master-0/wal/instance:
uuid: "ac05b5a84b9e431fa3c28dafcf775c5c"
format_stamp: "Formatted at 2026-09-29 22:54:15 on dist-test-slave-98mk"
server_key: "875d986e3f8f6a90e2dbf3bee417ebf9"
server_key_iv: "e74848311bfb8fd5b4591c3168c861de"
server_key_version: "encryptionkey@0"
I20260929 22:54:15.800100 2898 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260929 22:54:15.800859 2913 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 22:54:15.801040 2898 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260929 22:54:15.801124 2898 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790722213368738-14927-0/minicluster-data/master-0/data,/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790722213368738-14927-0/minicluster-data/master-0/wal
uuid: "ac05b5a84b9e431fa3c28dafcf775c5c"
format_stamp: "Formatted at 2026-09-29 22:54:15 on dist-test-slave-98mk"
server_key: "875d986e3f8f6a90e2dbf3bee417ebf9"
server_key_iv: "e74848311bfb8fd5b4591c3168c861de"
server_key_version: "encryptionkey@0"
I20260929 22:54:15.801182 2898 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790722213368738-14927-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790722213368738-14927-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790722213368738-14927-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 22:54:15.804443 2898 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 22:54:15.806564 2898 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 22:54:15.806696 2898 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 22:54:15.810339 2898 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:39345
I20260929 22:54:15.810381 2965 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:39345 every 8 connection(s)
I20260929 22:54:15.810715 2898 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790722213368738-14927-0/minicluster-data/master-0/data/info.pb
I20260929 22:54:15.811928 2966 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 22:54:15.814245 2966 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ac05b5a84b9e431fa3c28dafcf775c5c: Bootstrap starting.
I20260929 22:54:15.814831 2966 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ac05b5a84b9e431fa3c28dafcf775c5c: Neither blocks nor log segments found. Creating new log.
I20260929 22:54:15.815086 2966 log.cc:824] T 00000000000000000000000000000000 P ac05b5a84b9e431fa3c28dafcf775c5c: Log is configured to *not* fsync() on all Append() calls
I20260929 22:54:15.815331 14927 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhWOCLk/build/release/bin/kudu as pid 2898
I20260929 22:54:15.815467 14927 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790722213368738-14927-0/minicluster-data/master-0/wal/instance
I20260929 22:54:15.815615 14927 external_mini_cluster.cc:1428] Setting key ad77b24415a540bac8f1d994ce3dc1d3
I20260929 22:54:15.815836 2966 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ac05b5a84b9e431fa3c28dafcf775c5c: No bootstrap required, opened a new log
I20260929 22:54:15.817617 2966 raft_consensus.cc:348] T 00000000000000000000000000000000 P ac05b5a84b9e431fa3c28dafcf775c5c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ac05b5a84b9e431fa3c28dafcf775c5c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39345 } }
I20260929 22:54:15.817764 2966 raft_consensus.cc:374] T 00000000000000000000000000000000 P ac05b5a84b9e431fa3c28dafcf775c5c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 22:54:15.817795 2966 raft_consensus.cc:729] T 00000000000000000000000000000000 P ac05b5a84b9e431fa3c28dafcf775c5c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ac05b5a84b9e431fa3c28dafcf775c5c, State: Initialized, Role: FOLLOWER
I20260929 22:54:15.817919 2966 consensus_queue.cc:260] T 00000000000000000000000000000000 P ac05b5a84b9e431fa3c28dafcf775c5c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ac05b5a84b9e431fa3c28dafcf775c5c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39345 } }
I20260929 22:54:15.817996 2966 raft_consensus.cc:388] T 00000000000000000000000000000000 P ac05b5a84b9e431fa3c28dafcf775c5c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 22:54:15.818027 2966 raft_consensus.cc:482] T 00000000000000000000000000000000 P ac05b5a84b9e431fa3c28dafcf775c5c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 22:54:15.818104 2966 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ac05b5a84b9e431fa3c28dafcf775c5c [term 0 FOLLOWER]: Advancing to term 1
I20260929 22:54:15.818907 2970 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:54:15.816158 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60687 (local address 127.0.0.1:39345)
0929 22:54:15.816345 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0929 22:54:15.816353 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 22:54:15.816387 (+ 34us) server_negotiation.cc:407] Connection header received
0929 22:54:15.817627 (+ 1240us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 22:54:15.817638 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 22:54:15.817678 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 22:54:15.817833 (+ 155us) server_negotiation.cc:226] Negotiated authn=SASL
0929 22:54:15.818505 (+ 672us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 22:54:15.818509 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 22:54:15.818509 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 22:54:15.818513 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0929 22:54:15.818560 (+ 47us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 22:54:15.818571 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 22:54:15.818626 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0929 22:54:15.818684 (+ 58us) server_negotiation.cc:299] Negotiation successful
0929 22:54:15.818726 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":139,"thread_start_us":92,"threads_started":1}
I20260929 22:54:15.819144 2966 raft_consensus.cc:504] T 00000000000000000000000000000000 P ac05b5a84b9e431fa3c28dafcf775c5c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ac05b5a84b9e431fa3c28dafcf775c5c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39345 } }
I20260929 22:54:15.819254 2966 leader_election.cc:302] T 00000000000000000000000000000000 P ac05b5a84b9e431fa3c28dafcf775c5c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ac05b5a84b9e431fa3c28dafcf775c5c; no voters:
I20260929 22:54:15.819447 2966 leader_election.cc:288] T 00000000000000000000000000000000 P ac05b5a84b9e431fa3c28dafcf775c5c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 22:54:15.819492 2971 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ac05b5a84b9e431fa3c28dafcf775c5c [term 1 FOLLOWER]: Leader election won for term 1
I20260929 22:54:15.819638 2971 raft_consensus.cc:686] T 00000000000000000000000000000000 P ac05b5a84b9e431fa3c28dafcf775c5c [term 1 LEADER]: Becoming Leader. State: Replica: ac05b5a84b9e431fa3c28dafcf775c5c, State: Running, Role: LEADER
I20260929 22:54:15.819710 2966 sys_catalog.cc:564] T 00000000000000000000000000000000 P ac05b5a84b9e431fa3c28dafcf775c5c [sys.catalog]: configured and running, proceeding with master startup.
I20260929 22:54:15.819761 2971 consensus_queue.cc:237] T 00000000000000000000000000000000 P ac05b5a84b9e431fa3c28dafcf775c5c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ac05b5a84b9e431fa3c28dafcf775c5c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39345 } }
I20260929 22:54:15.819861 14927 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 22:54:15.820163 2971 sys_catalog.cc:455] T 00000000000000000000000000000000 P ac05b5a84b9e431fa3c28dafcf775c5c [sys.catalog]: SysCatalogTable state changed. Reason: New leader ac05b5a84b9e431fa3c28dafcf775c5c. Latest consensus state: current_term: 1 leader_uuid: "ac05b5a84b9e431fa3c28dafcf775c5c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ac05b5a84b9e431fa3c28dafcf775c5c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39345 } } }
I20260929 22:54:15.820246 2971 sys_catalog.cc:458] T 00000000000000000000000000000000 P ac05b5a84b9e431fa3c28dafcf775c5c [sys.catalog]: This master's current role is: LEADER
I20260929 22:54:15.820211 2972 sys_catalog.cc:455] T 00000000000000000000000000000000 P ac05b5a84b9e431fa3c28dafcf775c5c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ac05b5a84b9e431fa3c28dafcf775c5c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ac05b5a84b9e431fa3c28dafcf775c5c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39345 } } }
I20260929 22:54:15.820322 2972 sys_catalog.cc:458] T 00000000000000000000000000000000 P ac05b5a84b9e431fa3c28dafcf775c5c [sys.catalog]: This master's current role is: LEADER
I20260929 22:54:15.821067 2979 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 22:54:15.821624 2979 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 22:54:15.822525 2970 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:54:15.821396 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42983 (local address 127.0.0.1:39345)
0929 22:54:15.821444 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0929 22:54:15.821445 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0929 22:54:15.821582 (+ 137us) server_negotiation.cc:407] Connection header received
0929 22:54:15.821641 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 22:54:15.821641 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 22:54:15.821650 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 22:54:15.821701 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0929 22:54:15.822316 (+ 615us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 22:54:15.822317 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 22:54:15.822317 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 22:54:15.822321 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0929 22:54:15.822338 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 22:54:15.822343 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 22:54:15.822401 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0929 22:54:15.822457 (+ 56us) server_negotiation.cc:299] Negotiation successful
0929 22:54:15.822470 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260929 22:54:15.823215 2979 catalog_manager.cc:1347] Generated new cluster ID: d94507cf12734da59adc959a0b524658
I20260929 22:54:15.823264 2979 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 22:54:15.834002 2979 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 22:54:15.834293 2979 catalog_manager.cc:1504] Loading token signing keys...
I20260929 22:54:15.837410 2979 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ac05b5a84b9e431fa3c28dafcf775c5c: Generated new TSK 0
I20260929 22:54:15.837630 2979 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 22:54:15.841274 14927 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhWOCLk/build/release/bin/kudu with pid 2898
2026-09-29T22:54:15Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/2 (1203 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/3
Loading random data
Initializing database '/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790722213368738-14927-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 22:54:15 dist-test-slave-98mk krb5kdc[2997](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 22:54:15 dist-test-slave-98mk krb5kdc[2997](info): set up 2 sockets
Sep 29 22:54:15 dist-test-slave-98mk krb5kdc[2997](info): commencing operation
krb5kdc: starting...
W20260929 22:54:16.796185 14927 mini_kdc.cc:121] Time spent starting KDC: real 0.937s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790722213368738-14927-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790722213368738-14927-0/krb5kdc/test-user.keytab.
Sep 29 22:54:16 dist-test-slave-98mk krb5kdc[2997](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790722456, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T22:54:16Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T22:54:16Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790722213368738-14927-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790722213368738-14927-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260929 22:54:16.910019 14927 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhWOCLk/build/release/bin/kudu
/tmp/dist-test-taskhWOCLk/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790722213368738-14927-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790722213368738-14927-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790722213368738-14927-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790722213368738-14927-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37369
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790722213368738-14927-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:43757
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:37369
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790722213368738-14927-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790722213368738-14927-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790722213368738-14927-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 22:54:16.989820 3010 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 22:54:16.990042 3010 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 22:54:16.990067 3010 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 22:54:16.991288 3010 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 22:54:16.991333 3010 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 22:54:16.991346 3010 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 22:54:16.991358 3010 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 22:54:16.991369 3010 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 22:54:16.992915 3010 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:43757
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790722213368738-14927-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790722213368738-14927-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:37369
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790722213368738-14927-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37369
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790722213368738-14927-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3010
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790722213368738-14927-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 21:58:54 UTC on ccycloud.quasar-xguoti.root.comops.site
build id 4681390
I20260929 22:54:16.993166 3010 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 22:54:16.993388 3010 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 22:54:16.998354 3018 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 22:54:16.998359 3016 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 22:54:16.998373 3015 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 22:54:16.998600 3010 server_base.cc:1034] running on GCE node
I20260929 22:54:16.998823 3010 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 22:54:16.999130 3010 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 22:54:17.000277 3010 hybrid_clock.cc:648] HybridClock initialized: now 1790722457000262 us; error 30 us; skew 500 ppm
Sep 29 22:54:16 dist-test-slave-98mk krb5kdc[2997](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790722456, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 22:54:17.002719 3010 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260929 22:54:17.003151 3010 webserver.cc:460] Webserver started at http://127.0.0.1:37245/ using document root <none> and password file <none>
I20260929 22:54:17.003377 3010 fs_manager.cc:359] Metadata directory not provided
I20260929 22:54:17.003450 3010 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 22:54:17.003582 3010 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 22:54:17.004648 3010 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790722213368738-14927-0/minicluster-data/master-0/data/instance:
uuid: "82f33e9eb53146b9b47bb5d5e129f0fa"
format_stamp: "Formatted at 2026-09-29 22:54:17 on dist-test-slave-98mk"
server_key: "3a6e4c579a5b02425d2d2c7ea727cbb0"
server_key_iv: "ba7430f1051231b1b4390bec17bbbe4f"
server_key_version: "encryptionkey@0"
I20260929 22:54:17.004981 3010 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790722213368738-14927-0/minicluster-data/master-0/wal/instance:
uuid: "82f33e9eb53146b9b47bb5d5e129f0fa"
format_stamp: "Formatted at 2026-09-29 22:54:17 on dist-test-slave-98mk"
server_key: "3a6e4c579a5b02425d2d2c7ea727cbb0"
server_key_iv: "ba7430f1051231b1b4390bec17bbbe4f"
server_key_version: "encryptionkey@0"
I20260929 22:54:17.006338 3010 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260929 22:54:17.007148 3025 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 22:54:17.007303 3010 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260929 22:54:17.007386 3010 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790722213368738-14927-0/minicluster-data/master-0/data,/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790722213368738-14927-0/minicluster-data/master-0/wal
uuid: "82f33e9eb53146b9b47bb5d5e129f0fa"
format_stamp: "Formatted at 2026-09-29 22:54:17 on dist-test-slave-98mk"
server_key: "3a6e4c579a5b02425d2d2c7ea727cbb0"
server_key_iv: "ba7430f1051231b1b4390bec17bbbe4f"
server_key_version: "encryptionkey@0"
I20260929 22:54:17.007494 3010 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790722213368738-14927-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790722213368738-14927-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790722213368738-14927-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 22:54:17.017580 3010 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 22:54:17.019586 3010 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 22:54:17.019738 3010 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 22:54:17.023384 3010 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:37369
I20260929 22:54:17.023473 3077 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:37369 every 8 connection(s)
I20260929 22:54:17.023798 3010 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790722213368738-14927-0/minicluster-data/master-0/data/info.pb
I20260929 22:54:17.024955 14927 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhWOCLk/build/release/bin/kudu as pid 3010
I20260929 22:54:17.025019 3078 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 22:54:17.025102 14927 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790722213368738-14927-0/minicluster-data/master-0/wal/instance
I20260929 22:54:17.025449 14927 external_mini_cluster.cc:1428] Setting key 1044667db0712868770706548d0de19a
I20260929 22:54:17.028000 3078 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 82f33e9eb53146b9b47bb5d5e129f0fa: Bootstrap starting.
I20260929 22:54:17.028601 3078 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 82f33e9eb53146b9b47bb5d5e129f0fa: Neither blocks nor log segments found. Creating new log.
I20260929 22:54:17.028867 3078 log.cc:824] T 00000000000000000000000000000000 P 82f33e9eb53146b9b47bb5d5e129f0fa: Log is configured to *not* fsync() on all Append() calls
I20260929 22:54:17.029526 3078 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 82f33e9eb53146b9b47bb5d5e129f0fa: No bootstrap required, opened a new log
I20260929 22:54:17.031641 3078 raft_consensus.cc:348] T 00000000000000000000000000000000 P 82f33e9eb53146b9b47bb5d5e129f0fa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "82f33e9eb53146b9b47bb5d5e129f0fa" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37369 } }
I20260929 22:54:17.031791 3078 raft_consensus.cc:374] T 00000000000000000000000000000000 P 82f33e9eb53146b9b47bb5d5e129f0fa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 22:54:17.031826 3078 raft_consensus.cc:729] T 00000000000000000000000000000000 P 82f33e9eb53146b9b47bb5d5e129f0fa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 82f33e9eb53146b9b47bb5d5e129f0fa, State: Initialized, Role: FOLLOWER
I20260929 22:54:17.031968 3078 consensus_queue.cc:260] T 00000000000000000000000000000000 P 82f33e9eb53146b9b47bb5d5e129f0fa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "82f33e9eb53146b9b47bb5d5e129f0fa" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37369 } }
I20260929 22:54:17.032042 3078 raft_consensus.cc:388] T 00000000000000000000000000000000 P 82f33e9eb53146b9b47bb5d5e129f0fa [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 22:54:17.032081 3078 raft_consensus.cc:482] T 00000000000000000000000000000000 P 82f33e9eb53146b9b47bb5d5e129f0fa [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 22:54:17.032128 3078 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 82f33e9eb53146b9b47bb5d5e129f0fa [term 0 FOLLOWER]: Advancing to term 1
I20260929 22:54:17.032846 3078 raft_consensus.cc:504] T 00000000000000000000000000000000 P 82f33e9eb53146b9b47bb5d5e129f0fa [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "82f33e9eb53146b9b47bb5d5e129f0fa" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37369 } }
I20260929 22:54:17.032979 3078 leader_election.cc:302] T 00000000000000000000000000000000 P 82f33e9eb53146b9b47bb5d5e129f0fa [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 82f33e9eb53146b9b47bb5d5e129f0fa; no voters:
Sep 29 22:54:17 dist-test-slave-98mk krb5kdc[2997](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790722456, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260929 22:54:17.033174 3078 leader_election.cc:288] T 00000000000000000000000000000000 P 82f33e9eb53146b9b47bb5d5e129f0fa [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 22:54:17.033248 3083 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 82f33e9eb53146b9b47bb5d5e129f0fa [term 1 FOLLOWER]: Leader election won for term 1
I20260929 22:54:17.033455 3078 sys_catalog.cc:564] T 00000000000000000000000000000000 P 82f33e9eb53146b9b47bb5d5e129f0fa [sys.catalog]: configured and running, proceeding with master startup.
I20260929 22:54:17.033492 3083 raft_consensus.cc:686] T 00000000000000000000000000000000 P 82f33e9eb53146b9b47bb5d5e129f0fa [term 1 LEADER]: Becoming Leader. State: Replica: 82f33e9eb53146b9b47bb5d5e129f0fa, State: Running, Role: LEADER
I20260929 22:54:17.033660 3083 consensus_queue.cc:237] T 00000000000000000000000000000000 P 82f33e9eb53146b9b47bb5d5e129f0fa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "82f33e9eb53146b9b47bb5d5e129f0fa" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37369 } }
I20260929 22:54:17.034062 3084 sys_catalog.cc:455] T 00000000000000000000000000000000 P 82f33e9eb53146b9b47bb5d5e129f0fa [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "82f33e9eb53146b9b47bb5d5e129f0fa" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "82f33e9eb53146b9b47bb5d5e129f0fa" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37369 } } }
I20260929 22:54:17.034137 3084 sys_catalog.cc:458] T 00000000000000000000000000000000 P 82f33e9eb53146b9b47bb5d5e129f0fa [sys.catalog]: This master's current role is: LEADER
I20260929 22:54:17.034328 3085 sys_catalog.cc:455] T 00000000000000000000000000000000 P 82f33e9eb53146b9b47bb5d5e129f0fa [sys.catalog]: SysCatalogTable state changed. Reason: New leader 82f33e9eb53146b9b47bb5d5e129f0fa. Latest consensus state: current_term: 1 leader_uuid: "82f33e9eb53146b9b47bb5d5e129f0fa" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "82f33e9eb53146b9b47bb5d5e129f0fa" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37369 } } }
I20260929 22:54:17.034641 3085 sys_catalog.cc:458] T 00000000000000000000000000000000 P 82f33e9eb53146b9b47bb5d5e129f0fa [sys.catalog]: This master's current role is: LEADER
I20260929 22:54:17.034806 3092 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 22:54:17.035370 3092 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 22:54:17.037062 3092 catalog_manager.cc:1347] Generated new cluster ID: 7a9f0580b13a49f99a7dec680502faba
I20260929 22:54:17.037111 3092 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 22:54:17.037345 3081 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:54:17.026167 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51065 (local address 127.0.0.1:37369)
0929 22:54:17.026423 (+ 256us) server_negotiation.cc:206] Beginning negotiation
0929 22:54:17.026436 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0929 22:54:17.026472 (+ 36us) server_negotiation.cc:407] Connection header received
0929 22:54:17.027074 (+ 602us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 22:54:17.027082 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 22:54:17.027141 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 22:54:17.027301 (+ 160us) server_negotiation.cc:226] Negotiated authn=SASL
0929 22:54:17.028102 (+ 801us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:54:17.029254 (+ 1152us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:54:17.030072 (+ 818us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:54:17.030561 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:54:17.030687 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 22:54:17.033767 (+ 3080us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 22:54:17.033773 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 22:54:17.033773 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 22:54:17.033787 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0929 22:54:17.035753 (+ 1966us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 22:54:17.036138 (+ 385us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 22:54:17.036142 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 22:54:17.036143 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 22:54:17.036251 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 22:54:17.036454 (+ 203us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 22:54:17.036455 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 22:54:17.036455 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 22:54:17.036761 (+ 306us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 22:54:17.036898 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0929 22:54:17.037033 (+ 135us) server_negotiation.cc:299] Negotiation successful
0929 22:54:17.037177 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":184,"thread_start_us":114,"threads_started":1}
I20260929 22:54:17.040387 14927 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 22:54:17.053292 3081 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:54:17.041692 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34677 (local address 127.0.0.1:37369)
0929 22:54:17.041729 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0929 22:54:17.041731 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 22:54:17.047680 (+ 5949us) server_negotiation.cc:407] Connection header received
0929 22:54:17.047772 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 22:54:17.047774 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 22:54:17.047785 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 22:54:17.047814 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0929 22:54:17.048420 (+ 606us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:54:17.048921 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:54:17.049615 (+ 694us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:54:17.049843 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:54:17.049898 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 22:54:17.051021 (+ 1123us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 22:54:17.051023 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 22:54:17.051023 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 22:54:17.051031 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0929 22:54:17.052299 (+ 1268us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 22:54:17.052614 (+ 315us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 22:54:17.052615 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 22:54:17.052615 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 22:54:17.052664 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 22:54:17.052829 (+ 165us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 22:54:17.052830 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 22:54:17.052830 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 22:54:17.052936 (+ 106us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 22:54:17.053023 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0929 22:54:17.053124 (+ 101us) server_negotiation.cc:299] Negotiation successful
0929 22:54:17.053199 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260929 22:54:17.083025 3092 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 22:54:17.083802 3092 catalog_manager.cc:1504] Loading token signing keys...
I20260929 22:54:17.087848 3092 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 82f33e9eb53146b9b47bb5d5e129f0fa: Generated new TSK 0
I20260929 22:54:17.088048 3092 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 22:54:17.093408 14927 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhWOCLk/build/release/bin/kudu with pid 3010
2026-09-29T22:54:17Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/3 (1252 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/4
Loading random data
Initializing database '/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790722213368738-14927-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 22:54:17 dist-test-slave-98mk krb5kdc[3109](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 22:54:17 dist-test-slave-98mk krb5kdc[3109](info): set up 2 sockets
Sep 29 22:54:17 dist-test-slave-98mk krb5kdc[3109](info): commencing operation
krb5kdc: starting...
W20260929 22:54:18.052042 14927 mini_kdc.cc:121] Time spent starting KDC: real 0.941s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790722213368738-14927-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790722213368738-14927-0/krb5kdc/test-user.keytab.
Sep 29 22:54:18 dist-test-slave-98mk krb5kdc[3109](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790722458, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T22:54:18Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T22:54:18Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790722213368738-14927-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790722213368738-14927-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260929 22:54:18.166324 14927 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhWOCLk/build/release/bin/kudu
/tmp/dist-test-taskhWOCLk/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790722213368738-14927-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790722213368738-14927-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790722213368738-14927-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790722213368738-14927-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36375
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790722213368738-14927-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:44745
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:36375
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790722213368738-14927-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790722213368738-14927-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790722213368738-14927-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 22:54:18.248005 3122 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 22:54:18.248251 3122 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 22:54:18.248276 3122 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 22:54:18.249505 3122 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 22:54:18.249558 3122 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 22:54:18.249572 3122 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 22:54:18.249583 3122 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 22:54:18.249594 3122 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 22:54:18.251196 3122 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:44745
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790722213368738-14927-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790722213368738-14927-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:36375
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790722213368738-14927-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36375
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790722213368738-14927-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3122
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790722213368738-14927-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 21:58:54 UTC on ccycloud.quasar-xguoti.root.comops.site
build id 4681390
I20260929 22:54:18.251479 3122 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 22:54:18.251684 3122 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 22:54:18.256425 3127 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 22:54:18.256414 3128 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 22:54:18.256418 3122 server_base.cc:1034] running on GCE node
W20260929 22:54:18.256446 3130 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 22:54:18.256794 3122 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 22:54:18.257077 3122 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 22:54:18.258261 3122 hybrid_clock.cc:648] HybridClock initialized: now 1790722458258235 us; error 31 us; skew 500 ppm
Sep 29 22:54:18 dist-test-slave-98mk krb5kdc[3109](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790722458, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 22:54:18.261137 3122 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260929 22:54:18.261616 3122 webserver.cc:460] Webserver started at http://127.0.0.1:45789/ using document root <none> and password file <none>
I20260929 22:54:18.261823 3122 fs_manager.cc:359] Metadata directory not provided
I20260929 22:54:18.261873 3122 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 22:54:18.261999 3122 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 22:54:18.263084 3122 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790722213368738-14927-0/minicluster-data/master-0/data/instance:
uuid: "1f27f33b503d48ef8d47b75f749c0b2b"
format_stamp: "Formatted at 2026-09-29 22:54:18 on dist-test-slave-98mk"
server_key: "6471587a2d9e42e20707dba8b13c3ee0"
server_key_iv: "0a5e5283e44e334a8ff30d916b60deb6"
server_key_version: "encryptionkey@0"
I20260929 22:54:18.263463 3122 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790722213368738-14927-0/minicluster-data/master-0/wal/instance:
uuid: "1f27f33b503d48ef8d47b75f749c0b2b"
format_stamp: "Formatted at 2026-09-29 22:54:18 on dist-test-slave-98mk"
server_key: "6471587a2d9e42e20707dba8b13c3ee0"
server_key_iv: "0a5e5283e44e334a8ff30d916b60deb6"
server_key_version: "encryptionkey@0"
I20260929 22:54:18.264873 3122 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260929 22:54:18.265654 3137 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 22:54:18.265830 3122 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260929 22:54:18.265914 3122 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790722213368738-14927-0/minicluster-data/master-0/data,/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790722213368738-14927-0/minicluster-data/master-0/wal
uuid: "1f27f33b503d48ef8d47b75f749c0b2b"
format_stamp: "Formatted at 2026-09-29 22:54:18 on dist-test-slave-98mk"
server_key: "6471587a2d9e42e20707dba8b13c3ee0"
server_key_iv: "0a5e5283e44e334a8ff30d916b60deb6"
server_key_version: "encryptionkey@0"
I20260929 22:54:18.265983 3122 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790722213368738-14927-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790722213368738-14927-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790722213368738-14927-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 22:54:18.274106 3122 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 22:54:18.276346 3122 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 22:54:18.276501 3122 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 22:54:18.280373 3122 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:36375
I20260929 22:54:18.280416 3189 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:36375 every 8 connection(s)
I20260929 22:54:18.280747 3122 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790722213368738-14927-0/minicluster-data/master-0/data/info.pb
I20260929 22:54:18.281198 14927 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhWOCLk/build/release/bin/kudu as pid 3122
I20260929 22:54:18.281325 14927 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790722213368738-14927-0/minicluster-data/master-0/wal/instance
I20260929 22:54:18.281539 14927 external_mini_cluster.cc:1428] Setting key 4e5b725007b468c82d2df1829b1614ca
I20260929 22:54:18.281746 3190 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 22:54:18.284825 3190 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1f27f33b503d48ef8d47b75f749c0b2b: Bootstrap starting.
I20260929 22:54:18.285461 3190 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1f27f33b503d48ef8d47b75f749c0b2b: Neither blocks nor log segments found. Creating new log.
I20260929 22:54:18.285714 3190 log.cc:824] T 00000000000000000000000000000000 P 1f27f33b503d48ef8d47b75f749c0b2b: Log is configured to *not* fsync() on all Append() calls
I20260929 22:54:18.286394 3190 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1f27f33b503d48ef8d47b75f749c0b2b: No bootstrap required, opened a new log
I20260929 22:54:18.287009 3193 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:54:18.282164 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36709 (local address 127.0.0.1:36375)
0929 22:54:18.282324 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0929 22:54:18.282332 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 22:54:18.282355 (+ 23us) server_negotiation.cc:407] Connection header received
0929 22:54:18.282799 (+ 444us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 22:54:18.282805 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 22:54:18.282852 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 22:54:18.282963 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0929 22:54:18.283196 (+ 233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:54:18.284125 (+ 929us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:54:18.285199 (+ 1074us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:54:18.285652 (+ 453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:54:18.285737 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 22:54:18.286431 (+ 694us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 22:54:18.286436 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 22:54:18.286436 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 22:54:18.286608 (+ 172us) server_negotiation.cc:922] calling sasl_server_start()
0929 22:54:18.286638 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 22:54:18.286649 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 22:54:18.286701 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0929 22:54:18.286775 (+ 74us) server_negotiation.cc:299] Negotiation successful
0929 22:54:18.286847 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":116,"thread_start_us":67,"threads_started":1}
I20260929 22:54:18.287921 14927 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 22:54:18.288357 3190 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1f27f33b503d48ef8d47b75f749c0b2b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1f27f33b503d48ef8d47b75f749c0b2b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36375 } }
I20260929 22:54:18.288488 3190 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1f27f33b503d48ef8d47b75f749c0b2b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 22:54:18.288511 3190 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1f27f33b503d48ef8d47b75f749c0b2b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1f27f33b503d48ef8d47b75f749c0b2b, State: Initialized, Role: FOLLOWER
I20260929 22:54:18.288625 3190 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1f27f33b503d48ef8d47b75f749c0b2b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1f27f33b503d48ef8d47b75f749c0b2b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36375 } }
I20260929 22:54:18.288687 3190 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1f27f33b503d48ef8d47b75f749c0b2b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 22:54:18.288708 3190 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1f27f33b503d48ef8d47b75f749c0b2b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 22:54:18.288751 3190 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1f27f33b503d48ef8d47b75f749c0b2b [term 0 FOLLOWER]: Advancing to term 1
I20260929 22:54:18.289357 3190 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1f27f33b503d48ef8d47b75f749c0b2b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1f27f33b503d48ef8d47b75f749c0b2b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36375 } }
I20260929 22:54:18.289466 3190 leader_election.cc:302] T 00000000000000000000000000000000 P 1f27f33b503d48ef8d47b75f749c0b2b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1f27f33b503d48ef8d47b75f749c0b2b; no voters:
I20260929 22:54:18.289618 3190 leader_election.cc:288] T 00000000000000000000000000000000 P 1f27f33b503d48ef8d47b75f749c0b2b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 22:54:18.289649 3200 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1f27f33b503d48ef8d47b75f749c0b2b [term 1 FOLLOWER]: Leader election won for term 1
I20260929 22:54:18.289810 3200 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1f27f33b503d48ef8d47b75f749c0b2b [term 1 LEADER]: Becoming Leader. State: Replica: 1f27f33b503d48ef8d47b75f749c0b2b, State: Running, Role: LEADER
I20260929 22:54:18.289839 3190 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1f27f33b503d48ef8d47b75f749c0b2b [sys.catalog]: configured and running, proceeding with master startup.
I20260929 22:54:18.289904 3200 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1f27f33b503d48ef8d47b75f749c0b2b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1f27f33b503d48ef8d47b75f749c0b2b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36375 } }
I20260929 22:54:18.290200 3200 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1f27f33b503d48ef8d47b75f749c0b2b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1f27f33b503d48ef8d47b75f749c0b2b. Latest consensus state: current_term: 1 leader_uuid: "1f27f33b503d48ef8d47b75f749c0b2b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1f27f33b503d48ef8d47b75f749c0b2b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36375 } } }
I20260929 22:54:18.290261 3201 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1f27f33b503d48ef8d47b75f749c0b2b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1f27f33b503d48ef8d47b75f749c0b2b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1f27f33b503d48ef8d47b75f749c0b2b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36375 } } }
I20260929 22:54:18.290333 3200 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1f27f33b503d48ef8d47b75f749c0b2b [sys.catalog]: This master's current role is: LEADER
I20260929 22:54:18.290388 3201 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1f27f33b503d48ef8d47b75f749c0b2b [sys.catalog]: This master's current role is: LEADER
W20260929 22:54:18.291695 3214 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 1f27f33b503d48ef8d47b75f749c0b2b: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 22:54:18.292045 3214 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260929 22:54:18.292147 3215 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 22:54:18.292304 3215 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 22:54:18.292975 3193 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:54:18.289207 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35649 (local address 127.0.0.1:36375)
0929 22:54:18.289247 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0929 22:54:18.289248 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0929 22:54:18.289296 (+ 48us) server_negotiation.cc:407] Connection header received
0929 22:54:18.289344 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 22:54:18.289345 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 22:54:18.289355 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 22:54:18.289428 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0929 22:54:18.289543 (+ 115us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:54:18.290220 (+ 677us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:54:18.291087 (+ 867us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:54:18.291394 (+ 307us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:54:18.291482 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 22:54:18.292733 (+ 1251us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 22:54:18.292734 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 22:54:18.292734 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 22:54:18.292738 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0929 22:54:18.292751 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 22:54:18.292754 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 22:54:18.292801 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0929 22:54:18.292874 (+ 73us) server_negotiation.cc:299] Negotiation successful
0929 22:54:18.292898 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260929 22:54:18.293666 3215 catalog_manager.cc:1347] Generated new cluster ID: 8a82d716240f477a9bf3cb98d5890526
I20260929 22:54:18.293715 3215 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 22:54:18.299305 3215 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 22:54:18.299820 3215 catalog_manager.cc:1504] Loading token signing keys...
I20260929 22:54:18.304121 3215 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1f27f33b503d48ef8d47b75f749c0b2b: Generated new TSK 0
I20260929 22:54:18.304278 3215 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 22:54:18.309911 14927 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhWOCLk/build/release/bin/kudu with pid 3122
2026-09-29T22:54:18Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/4 (1216 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/5
Loading random data
Initializing database '/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790722213368738-14927-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 22:54:18 dist-test-slave-98mk krb5kdc[3221](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 22:54:18 dist-test-slave-98mk krb5kdc[3221](info): set up 2 sockets
Sep 29 22:54:18 dist-test-slave-98mk krb5kdc[3221](info): commencing operation
krb5kdc: starting...
W20260929 22:54:19.268023 14927 mini_kdc.cc:121] Time spent starting KDC: real 0.941s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790722213368738-14927-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790722213368738-14927-0/krb5kdc/test-user.keytab.
Sep 29 22:54:19 dist-test-slave-98mk krb5kdc[3221](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790722459, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T22:54:19Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T22:54:19Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790722213368738-14927-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790722213368738-14927-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260929 22:54:19.384109 14927 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhWOCLk/build/release/bin/kudu
/tmp/dist-test-taskhWOCLk/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790722213368738-14927-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790722213368738-14927-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790722213368738-14927-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790722213368738-14927-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38209
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790722213368738-14927-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:36167
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:38209
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790722213368738-14927-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790722213368738-14927-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790722213368738-14927-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 22:54:19.465936 3234 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 22:54:19.466181 3234 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 22:54:19.466215 3234 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 22:54:19.467510 3234 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 22:54:19.467566 3234 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 22:54:19.467584 3234 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 22:54:19.467602 3234 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 22:54:19.467617 3234 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 22:54:19.469277 3234 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:36167
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790722213368738-14927-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790722213368738-14927-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:38209
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790722213368738-14927-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38209
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790722213368738-14927-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3234
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790722213368738-14927-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 21:58:54 UTC on ccycloud.quasar-xguoti.root.comops.site
build id 4681390
I20260929 22:54:19.469568 3234 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 22:54:19.469794 3234 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 22:54:19.474696 3240 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 22:54:19.474686 3242 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 22:54:19.474720 3234 server_base.cc:1034] running on GCE node
W20260929 22:54:19.474689 3239 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 22:54:19.475124 3234 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 22:54:19.475443 3234 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 22:54:19.476601 3234 hybrid_clock.cc:648] HybridClock initialized: now 1790722459476569 us; error 51 us; skew 500 ppm
Sep 29 22:54:19 dist-test-slave-98mk krb5kdc[3221](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790722459, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 22:54:19.479153 3234 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260929 22:54:19.479640 3234 webserver.cc:460] Webserver started at http://127.0.0.1:36725/ using document root <none> and password file <none>
I20260929 22:54:19.479851 3234 fs_manager.cc:359] Metadata directory not provided
I20260929 22:54:19.479893 3234 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 22:54:19.479992 3234 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 22:54:19.481099 3234 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790722213368738-14927-0/minicluster-data/master-0/data/instance:
uuid: "19b298427e8f4c23b7beb691b6767665"
format_stamp: "Formatted at 2026-09-29 22:54:19 on dist-test-slave-98mk"
server_key: "0ad476904f50d22c3a16f165f1957c66"
server_key_iv: "2d7023d1fc658851d4d9fa864fd6480a"
server_key_version: "encryptionkey@0"
I20260929 22:54:19.481403 3234 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790722213368738-14927-0/minicluster-data/master-0/wal/instance:
uuid: "19b298427e8f4c23b7beb691b6767665"
format_stamp: "Formatted at 2026-09-29 22:54:19 on dist-test-slave-98mk"
server_key: "0ad476904f50d22c3a16f165f1957c66"
server_key_iv: "2d7023d1fc658851d4d9fa864fd6480a"
server_key_version: "encryptionkey@0"
I20260929 22:54:19.482645 3234 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260929 22:54:19.483481 3249 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 22:54:19.483639 3234 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260929 22:54:19.483734 3234 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790722213368738-14927-0/minicluster-data/master-0/data,/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790722213368738-14927-0/minicluster-data/master-0/wal
uuid: "19b298427e8f4c23b7beb691b6767665"
format_stamp: "Formatted at 2026-09-29 22:54:19 on dist-test-slave-98mk"
server_key: "0ad476904f50d22c3a16f165f1957c66"
server_key_iv: "2d7023d1fc658851d4d9fa864fd6480a"
server_key_version: "encryptionkey@0"
I20260929 22:54:19.483798 3234 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790722213368738-14927-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790722213368738-14927-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790722213368738-14927-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 22:54:19.487104 3234 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 22:54:19.488955 3234 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 22:54:19.489064 3234 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 22:54:19.492738 3234 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:38209
I20260929 22:54:19.492772 3301 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:38209 every 8 connection(s)
I20260929 22:54:19.493089 3234 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790722213368738-14927-0/minicluster-data/master-0/data/info.pb
I20260929 22:54:19.494009 3302 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 22:54:19.496374 3302 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 19b298427e8f4c23b7beb691b6767665: Bootstrap starting.
I20260929 22:54:19.496939 3302 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 19b298427e8f4c23b7beb691b6767665: Neither blocks nor log segments found. Creating new log.
I20260929 22:54:19.497188 3302 log.cc:824] T 00000000000000000000000000000000 P 19b298427e8f4c23b7beb691b6767665: Log is configured to *not* fsync() on all Append() calls
I20260929 22:54:19.497848 3302 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 19b298427e8f4c23b7beb691b6767665: No bootstrap required, opened a new log
I20260929 22:54:19.499104 14927 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhWOCLk/build/release/bin/kudu as pid 3234
I20260929 22:54:19.499200 14927 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790722213368738-14927-0/minicluster-data/master-0/wal/instance
I20260929 22:54:19.499399 14927 external_mini_cluster.cc:1428] Setting key 20fe5cba657af806103cdb4fdbbf564c
I20260929 22:54:19.499574 3302 raft_consensus.cc:348] T 00000000000000000000000000000000 P 19b298427e8f4c23b7beb691b6767665 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "19b298427e8f4c23b7beb691b6767665" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38209 } }
I20260929 22:54:19.499761 3302 raft_consensus.cc:374] T 00000000000000000000000000000000 P 19b298427e8f4c23b7beb691b6767665 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 22:54:19.499850 3302 raft_consensus.cc:729] T 00000000000000000000000000000000 P 19b298427e8f4c23b7beb691b6767665 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 19b298427e8f4c23b7beb691b6767665, State: Initialized, Role: FOLLOWER
I20260929 22:54:19.500001 3302 consensus_queue.cc:260] T 00000000000000000000000000000000 P 19b298427e8f4c23b7beb691b6767665 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "19b298427e8f4c23b7beb691b6767665" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38209 } }
I20260929 22:54:19.500078 3302 raft_consensus.cc:388] T 00000000000000000000000000000000 P 19b298427e8f4c23b7beb691b6767665 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 22:54:19.500109 3302 raft_consensus.cc:482] T 00000000000000000000000000000000 P 19b298427e8f4c23b7beb691b6767665 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 22:54:19.500173 3302 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 19b298427e8f4c23b7beb691b6767665 [term 0 FOLLOWER]: Advancing to term 1
I20260929 22:54:19.501176 3302 raft_consensus.cc:504] T 00000000000000000000000000000000 P 19b298427e8f4c23b7beb691b6767665 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "19b298427e8f4c23b7beb691b6767665" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38209 } }
I20260929 22:54:19.501279 3302 leader_election.cc:302] T 00000000000000000000000000000000 P 19b298427e8f4c23b7beb691b6767665 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 19b298427e8f4c23b7beb691b6767665; no voters:
I20260929 22:54:19.501454 3302 leader_election.cc:288] T 00000000000000000000000000000000 P 19b298427e8f4c23b7beb691b6767665 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 22:54:19.501605 3308 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 19b298427e8f4c23b7beb691b6767665 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 22:54:19.501665 3306 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:54:19.499973 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48877 (local address 127.0.0.1:38209)
0929 22:54:19.500185 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0929 22:54:19.500194 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 22:54:19.500229 (+ 35us) server_negotiation.cc:407] Connection header received
0929 22:54:19.500348 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 22:54:19.500354 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 22:54:19.500376 (+ 22us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 22:54:19.500506 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0929 22:54:19.501190 (+ 684us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 22:54:19.501194 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 22:54:19.501194 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 22:54:19.501198 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0929 22:54:19.501241 (+ 43us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 22:54:19.501252 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 22:54:19.501313 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0929 22:54:19.501426 (+ 113us) server_negotiation.cc:299] Negotiation successful
0929 22:54:19.501472 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":162,"thread_start_us":86,"threads_started":1}
I20260929 22:54:19.501696 3302 sys_catalog.cc:564] T 00000000000000000000000000000000 P 19b298427e8f4c23b7beb691b6767665 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 22:54:19.501734 3308 raft_consensus.cc:686] T 00000000000000000000000000000000 P 19b298427e8f4c23b7beb691b6767665 [term 1 LEADER]: Becoming Leader. State: Replica: 19b298427e8f4c23b7beb691b6767665, State: Running, Role: LEADER
I20260929 22:54:19.502388 3308 consensus_queue.cc:237] T 00000000000000000000000000000000 P 19b298427e8f4c23b7beb691b6767665 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "19b298427e8f4c23b7beb691b6767665" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38209 } }
I20260929 22:54:19.502722 14927 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 22:54:19.503170 3307 sys_catalog.cc:455] T 00000000000000000000000000000000 P 19b298427e8f4c23b7beb691b6767665 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "19b298427e8f4c23b7beb691b6767665" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "19b298427e8f4c23b7beb691b6767665" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38209 } } }
I20260929 22:54:19.503170 3309 sys_catalog.cc:455] T 00000000000000000000000000000000 P 19b298427e8f4c23b7beb691b6767665 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 19b298427e8f4c23b7beb691b6767665. Latest consensus state: current_term: 1 leader_uuid: "19b298427e8f4c23b7beb691b6767665" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "19b298427e8f4c23b7beb691b6767665" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38209 } } }
I20260929 22:54:19.503626 3307 sys_catalog.cc:458] T 00000000000000000000000000000000 P 19b298427e8f4c23b7beb691b6767665 [sys.catalog]: This master's current role is: LEADER
I20260929 22:54:19.503706 3309 sys_catalog.cc:458] T 00000000000000000000000000000000 P 19b298427e8f4c23b7beb691b6767665 [sys.catalog]: This master's current role is: LEADER
W20260929 22:54:19.504156 3321 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 19b298427e8f4c23b7beb691b6767665: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 22:54:19.504251 3321 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260929 22:54:19.504359 3325 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 22:54:19.504559 3325 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 22:54:19.505347 3306 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:54:19.504254 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54859 (local address 127.0.0.1:38209)
0929 22:54:19.504288 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0929 22:54:19.504290 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 22:54:19.504452 (+ 162us) server_negotiation.cc:407] Connection header received
0929 22:54:19.504502 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 22:54:19.504503 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 22:54:19.504508 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 22:54:19.504572 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0929 22:54:19.505120 (+ 548us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 22:54:19.505122 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 22:54:19.505122 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 22:54:19.505126 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0929 22:54:19.505144 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 22:54:19.505148 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 22:54:19.505205 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0929 22:54:19.505273 (+ 68us) server_negotiation.cc:299] Negotiation successful
0929 22:54:19.505286 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260929 22:54:19.506093 3325 catalog_manager.cc:1347] Generated new cluster ID: c96c3422a452426681b5e83f2bf41b77
I20260929 22:54:19.506142 3325 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 22:54:19.524681 3325 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 22:54:19.524853 3325 catalog_manager.cc:1504] Loading token signing keys...
I20260929 22:54:19.528559 3325 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 19b298427e8f4c23b7beb691b6767665: Generated new TSK 0
I20260929 22:54:19.528728 3325 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 22:54:19.530754 14927 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhWOCLk/build/release/bin/kudu with pid 3234
2026-09-29T22:54:19Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/5 (1220 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/6
Loading random data
Initializing database '/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790722213368738-14927-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 22:54:19 dist-test-slave-98mk krb5kdc[3333](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 22:54:19 dist-test-slave-98mk krb5kdc[3333](info): set up 2 sockets
Sep 29 22:54:19 dist-test-slave-98mk krb5kdc[3333](info): commencing operation
krb5kdc: starting...
W20260929 22:54:20.484056 14927 mini_kdc.cc:121] Time spent starting KDC: real 0.936s user 0.005s sys 0.000s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790722213368738-14927-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790722213368738-14927-0/krb5kdc/test-user.keytab.
Sep 29 22:54:20 dist-test-slave-98mk krb5kdc[3333](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790722460, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T22:54:20Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T22:54:20Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790722213368738-14927-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790722213368738-14927-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260929 22:54:20.599900 14927 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhWOCLk/build/release/bin/kudu
/tmp/dist-test-taskhWOCLk/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790722213368738-14927-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790722213368738-14927-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790722213368738-14927-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790722213368738-14927-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42797
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790722213368738-14927-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:36445
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:42797
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790722213368738-14927-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790722213368738-14927-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790722213368738-14927-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 22:54:20.681121 3346 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 22:54:20.681376 3346 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 22:54:20.681411 3346 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 22:54:20.682668 3346 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 22:54:20.682720 3346 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 22:54:20.682739 3346 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 22:54:20.682755 3346 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 22:54:20.682770 3346 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 22:54:20.684423 3346 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:36445
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790722213368738-14927-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790722213368738-14927-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:42797
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790722213368738-14927-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42797
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790722213368738-14927-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3346
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790722213368738-14927-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 21:58:54 UTC on ccycloud.quasar-xguoti.root.comops.site
build id 4681390
I20260929 22:54:20.684695 3346 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 22:54:20.684917 3346 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 22:54:20.689597 3351 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 22:54:20.689574 3354 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 22:54:20.689569 3352 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 22:54:20.689781 3346 server_base.cc:1034] running on GCE node
I20260929 22:54:20.690057 3346 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 22:54:20.690356 3346 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 22:54:20.691475 3346 hybrid_clock.cc:648] HybridClock initialized: now 1790722460691482 us; error 56 us; skew 500 ppm
Sep 29 22:54:20 dist-test-slave-98mk krb5kdc[3333](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790722460, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 22:54:20.698534 3346 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260929 22:54:20.699002 3346 webserver.cc:460] Webserver started at http://127.0.0.1:39891/ using document root <none> and password file <none>
I20260929 22:54:20.699231 3346 fs_manager.cc:359] Metadata directory not provided
I20260929 22:54:20.699282 3346 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 22:54:20.699441 3346 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 22:54:20.700538 3346 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790722213368738-14927-0/minicluster-data/master-0/data/instance:
uuid: "5e90a922ecd7403f8270d9717b60e13b"
format_stamp: "Formatted at 2026-09-29 22:54:20 on dist-test-slave-98mk"
server_key: "c1391c0130c127fea4ccf1e7781684d3"
server_key_iv: "7b20909037f4c85bf44414730257ebf0"
server_key_version: "encryptionkey@0"
I20260929 22:54:20.700875 3346 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790722213368738-14927-0/minicluster-data/master-0/wal/instance:
uuid: "5e90a922ecd7403f8270d9717b60e13b"
format_stamp: "Formatted at 2026-09-29 22:54:20 on dist-test-slave-98mk"
server_key: "c1391c0130c127fea4ccf1e7781684d3"
server_key_iv: "7b20909037f4c85bf44414730257ebf0"
server_key_version: "encryptionkey@0"
I20260929 22:54:20.702168 3346 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260929 22:54:20.702956 3361 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 22:54:20.703145 3346 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260929 22:54:20.703227 3346 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790722213368738-14927-0/minicluster-data/master-0/data,/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790722213368738-14927-0/minicluster-data/master-0/wal
uuid: "5e90a922ecd7403f8270d9717b60e13b"
format_stamp: "Formatted at 2026-09-29 22:54:20 on dist-test-slave-98mk"
server_key: "c1391c0130c127fea4ccf1e7781684d3"
server_key_iv: "7b20909037f4c85bf44414730257ebf0"
server_key_version: "encryptionkey@0"
I20260929 22:54:20.703299 3346 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790722213368738-14927-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790722213368738-14927-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790722213368738-14927-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 22:54:20.713307 3346 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 22:54:20.715406 3346 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 22:54:20.715638 3346 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 22:54:20.719246 3346 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:42797
I20260929 22:54:20.719282 3413 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:42797 every 8 connection(s)
I20260929 22:54:20.719678 3346 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790722213368738-14927-0/minicluster-data/master-0/data/info.pb
I20260929 22:54:20.720546 3414 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 22:54:20.722798 3414 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5e90a922ecd7403f8270d9717b60e13b: Bootstrap starting.
I20260929 22:54:20.723361 3414 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5e90a922ecd7403f8270d9717b60e13b: Neither blocks nor log segments found. Creating new log.
I20260929 22:54:20.723664 3414 log.cc:824] T 00000000000000000000000000000000 P 5e90a922ecd7403f8270d9717b60e13b: Log is configured to *not* fsync() on all Append() calls
I20260929 22:54:20.724360 3414 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5e90a922ecd7403f8270d9717b60e13b: No bootstrap required, opened a new log
I20260929 22:54:20.724869 14927 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhWOCLk/build/release/bin/kudu as pid 3346
I20260929 22:54:20.724992 14927 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790722213368738-14927-0/minicluster-data/master-0/wal/instance
I20260929 22:54:20.725155 14927 external_mini_cluster.cc:1428] Setting key eb13362b1aeb0dd48ee6dbcd523caef9
I20260929 22:54:20.726475 3414 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5e90a922ecd7403f8270d9717b60e13b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5e90a922ecd7403f8270d9717b60e13b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42797 } }
I20260929 22:54:20.726683 3414 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5e90a922ecd7403f8270d9717b60e13b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 22:54:20.726717 3414 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5e90a922ecd7403f8270d9717b60e13b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5e90a922ecd7403f8270d9717b60e13b, State: Initialized, Role: FOLLOWER
I20260929 22:54:20.726840 3414 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5e90a922ecd7403f8270d9717b60e13b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5e90a922ecd7403f8270d9717b60e13b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42797 } }
I20260929 22:54:20.726918 3414 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5e90a922ecd7403f8270d9717b60e13b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 22:54:20.726948 3414 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5e90a922ecd7403f8270d9717b60e13b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 22:54:20.726994 3414 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5e90a922ecd7403f8270d9717b60e13b [term 0 FOLLOWER]: Advancing to term 1
I20260929 22:54:20.727636 3414 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5e90a922ecd7403f8270d9717b60e13b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5e90a922ecd7403f8270d9717b60e13b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42797 } }
I20260929 22:54:20.727744 3414 leader_election.cc:302] T 00000000000000000000000000000000 P 5e90a922ecd7403f8270d9717b60e13b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5e90a922ecd7403f8270d9717b60e13b; no voters:
I20260929 22:54:20.727896 3414 leader_election.cc:288] T 00000000000000000000000000000000 P 5e90a922ecd7403f8270d9717b60e13b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 22:54:20.727942 3419 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5e90a922ecd7403f8270d9717b60e13b [term 1 FOLLOWER]: Leader election won for term 1
I20260929 22:54:20.728096 3419 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5e90a922ecd7403f8270d9717b60e13b [term 1 LEADER]: Becoming Leader. State: Replica: 5e90a922ecd7403f8270d9717b60e13b, State: Running, Role: LEADER
I20260929 22:54:20.728170 3414 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5e90a922ecd7403f8270d9717b60e13b [sys.catalog]: configured and running, proceeding with master startup.
I20260929 22:54:20.728206 3419 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5e90a922ecd7403f8270d9717b60e13b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5e90a922ecd7403f8270d9717b60e13b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42797 } }
I20260929 22:54:20.728552 3419 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5e90a922ecd7403f8270d9717b60e13b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5e90a922ecd7403f8270d9717b60e13b. Latest consensus state: current_term: 1 leader_uuid: "5e90a922ecd7403f8270d9717b60e13b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5e90a922ecd7403f8270d9717b60e13b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42797 } } }
I20260929 22:54:20.728739 3419 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5e90a922ecd7403f8270d9717b60e13b [sys.catalog]: This master's current role is: LEADER
I20260929 22:54:20.728682 3420 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5e90a922ecd7403f8270d9717b60e13b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5e90a922ecd7403f8270d9717b60e13b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5e90a922ecd7403f8270d9717b60e13b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42797 } } }
I20260929 22:54:20.728982 3420 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5e90a922ecd7403f8270d9717b60e13b [sys.catalog]: This master's current role is: LEADER
I20260929 22:54:20.729351 3428 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 22:54:20.729895 3428 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 22:54:20.731468 3428 catalog_manager.cc:1347] Generated new cluster ID: 7f72d21b72ce4d25a127e74bd6f9c919
I20260929 22:54:20.731527 3428 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 29 22:54:20 dist-test-slave-98mk krb5kdc[3333](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790722460, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260929 22:54:20.737447 3418 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:54:20.725675 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.71:60641 (local address 127.0.0.1:42797)
0929 22:54:20.725870 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0929 22:54:20.725880 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 22:54:20.725921 (+ 41us) server_negotiation.cc:407] Connection header received
0929 22:54:20.726644 (+ 723us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 22:54:20.726653 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 22:54:20.726721 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 22:54:20.726871 (+ 150us) server_negotiation.cc:226] Negotiated authn=SASL
0929 22:54:20.727542 (+ 671us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:54:20.728954 (+ 1412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:54:20.730001 (+ 1047us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:54:20.730574 (+ 573us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:54:20.730673 (+ 99us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 22:54:20.733865 (+ 3192us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 22:54:20.733870 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 22:54:20.733870 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 22:54:20.733882 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0929 22:54:20.735835 (+ 1953us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 22:54:20.736287 (+ 452us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 22:54:20.736289 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 22:54:20.736290 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 22:54:20.736388 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 22:54:20.736597 (+ 209us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 22:54:20.736598 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 22:54:20.736598 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 22:54:20.736881 (+ 283us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 22:54:20.737045 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0929 22:54:20.737152 (+ 107us) server_negotiation.cc:299] Negotiation successful
0929 22:54:20.737249 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":139,"thread_start_us":93,"threads_started":1}
I20260929 22:54:20.738586 14927 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 22:54:20.749707 3428 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 22:54:20.750226 3428 catalog_manager.cc:1504] Loading token signing keys...
I20260929 22:54:20.755155 3418 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:54:20.740127 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.71:48149 (local address 127.0.0.1:42797)
0929 22:54:20.740164 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0929 22:54:20.740166 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 22:54:20.747642 (+ 7476us) server_negotiation.cc:407] Connection header received
0929 22:54:20.747726 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 22:54:20.747728 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 22:54:20.747742 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 22:54:20.747816 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0929 22:54:20.748524 (+ 708us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:54:20.749351 (+ 827us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:54:20.750435 (+ 1084us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:54:20.750752 (+ 317us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:54:20.750826 (+ 74us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 22:54:20.752447 (+ 1621us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 22:54:20.752449 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 22:54:20.752449 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 22:54:20.752457 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0929 22:54:20.753959 (+ 1502us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 22:54:20.754394 (+ 435us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 22:54:20.754395 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 22:54:20.754396 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 22:54:20.754446 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 22:54:20.754641 (+ 195us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 22:54:20.754641 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 22:54:20.754641 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 22:54:20.754755 (+ 114us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 22:54:20.754856 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0929 22:54:20.754976 (+ 120us) server_negotiation.cc:299] Negotiation successful
0929 22:54:20.755047 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260929 22:54:20.756261 3428 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5e90a922ecd7403f8270d9717b60e13b: Generated new TSK 0
I20260929 22:54:20.756464 3428 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 22:54:20.762889 14927 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhWOCLk/build/release/bin/kudu with pid 3346
W20260929 22:54:20.766584 3331 connection.cc:535] client connection to 127.0.0.1:42797 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:42797): Connection reset by peer (error 104)
2026-09-29T22:54:20Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/6 (1231 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/7
Loading random data
Initializing database '/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790722213368738-14927-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 22:54:20 dist-test-slave-98mk krb5kdc[3445](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 22:54:20 dist-test-slave-98mk krb5kdc[3445](info): set up 2 sockets
Sep 29 22:54:20 dist-test-slave-98mk krb5kdc[3445](info): commencing operation
krb5kdc: starting...
W20260929 22:54:21.700011 14927 mini_kdc.cc:121] Time spent starting KDC: real 0.920s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790722213368738-14927-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790722213368738-14927-0/krb5kdc/test-user.keytab.
Sep 29 22:54:21 dist-test-slave-98mk krb5kdc[3445](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790722461, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T22:54:21Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T22:54:21Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790722213368738-14927-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790722213368738-14927-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260929 22:54:21.818039 14927 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhWOCLk/build/release/bin/kudu
/tmp/dist-test-taskhWOCLk/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790722213368738-14927-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790722213368738-14927-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790722213368738-14927-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790722213368738-14927-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43939
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790722213368738-14927-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:37665
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:43939
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790722213368738-14927-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790722213368738-14927-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790722213368738-14927-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 22:54:21.899443 3458 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 22:54:21.899686 3458 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 22:54:21.899718 3458 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 22:54:21.901067 3458 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 22:54:21.901136 3458 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 22:54:21.901160 3458 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 22:54:21.901177 3458 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 22:54:21.901192 3458 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 22:54:21.902781 3458 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:37665
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790722213368738-14927-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790722213368738-14927-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:43939
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790722213368738-14927-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43939
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790722213368738-14927-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3458
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790722213368738-14927-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 21:58:54 UTC on ccycloud.quasar-xguoti.root.comops.site
build id 4681390
I20260929 22:54:21.903028 3458 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 22:54:21.903246 3458 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 22:54:21.907809 3464 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 22:54:21.907859 3458 server_base.cc:1034] running on GCE node
W20260929 22:54:21.907794 3466 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 22:54:21.907794 3463 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 22:54:21.908211 3458 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 22:54:21.908500 3458 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 22:54:21.909633 3458 hybrid_clock.cc:648] HybridClock initialized: now 1790722461909621 us; error 27 us; skew 500 ppm
Sep 29 22:54:21 dist-test-slave-98mk krb5kdc[3445](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790722461, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 22:54:21.911938 3458 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260929 22:54:21.912322 3458 webserver.cc:460] Webserver started at http://127.0.0.1:39631/ using document root <none> and password file <none>
I20260929 22:54:21.912545 3458 fs_manager.cc:359] Metadata directory not provided
I20260929 22:54:21.912592 3458 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 22:54:21.912686 3458 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 22:54:21.913699 3458 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790722213368738-14927-0/minicluster-data/master-0/data/instance:
uuid: "a39461715cd242589ae37187c4c048f6"
format_stamp: "Formatted at 2026-09-29 22:54:21 on dist-test-slave-98mk"
server_key: "a9a6b1f3bb6ffa52707fa6e472f9c10f"
server_key_iv: "d84bf578df24173a927928670057d679"
server_key_version: "encryptionkey@0"
I20260929 22:54:21.913991 3458 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790722213368738-14927-0/minicluster-data/master-0/wal/instance:
uuid: "a39461715cd242589ae37187c4c048f6"
format_stamp: "Formatted at 2026-09-29 22:54:21 on dist-test-slave-98mk"
server_key: "a9a6b1f3bb6ffa52707fa6e472f9c10f"
server_key_iv: "d84bf578df24173a927928670057d679"
server_key_version: "encryptionkey@0"
I20260929 22:54:21.915164 3458 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260929 22:54:21.915846 3473 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 22:54:21.915992 3458 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260929 22:54:21.916072 3458 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790722213368738-14927-0/minicluster-data/master-0/data,/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790722213368738-14927-0/minicluster-data/master-0/wal
uuid: "a39461715cd242589ae37187c4c048f6"
format_stamp: "Formatted at 2026-09-29 22:54:21 on dist-test-slave-98mk"
server_key: "a9a6b1f3bb6ffa52707fa6e472f9c10f"
server_key_iv: "d84bf578df24173a927928670057d679"
server_key_version: "encryptionkey@0"
I20260929 22:54:21.916128 3458 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790722213368738-14927-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790722213368738-14927-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790722213368738-14927-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 22:54:21.939499 3458 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 22:54:21.941412 3458 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 22:54:21.941526 3458 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 22:54:21.945218 3458 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:43939
I20260929 22:54:21.945262 3525 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:43939 every 8 connection(s)
I20260929 22:54:21.945582 3458 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790722213368738-14927-0/minicluster-data/master-0/data/info.pb
I20260929 22:54:21.946403 3526 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 22:54:21.948716 3526 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a39461715cd242589ae37187c4c048f6: Bootstrap starting.
I20260929 22:54:21.949262 3526 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a39461715cd242589ae37187c4c048f6: Neither blocks nor log segments found. Creating new log.
I20260929 22:54:21.949522 3526 log.cc:824] T 00000000000000000000000000000000 P a39461715cd242589ae37187c4c048f6: Log is configured to *not* fsync() on all Append() calls
I20260929 22:54:21.950158 3526 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a39461715cd242589ae37187c4c048f6: No bootstrap required, opened a new log
I20260929 22:54:21.951932 3526 raft_consensus.cc:348] T 00000000000000000000000000000000 P a39461715cd242589ae37187c4c048f6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a39461715cd242589ae37187c4c048f6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43939 } }
I20260929 22:54:21.952114 3526 raft_consensus.cc:374] T 00000000000000000000000000000000 P a39461715cd242589ae37187c4c048f6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 22:54:21.952149 3526 raft_consensus.cc:729] T 00000000000000000000000000000000 P a39461715cd242589ae37187c4c048f6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a39461715cd242589ae37187c4c048f6, State: Initialized, Role: FOLLOWER
I20260929 22:54:21.952294 3526 consensus_queue.cc:260] T 00000000000000000000000000000000 P a39461715cd242589ae37187c4c048f6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a39461715cd242589ae37187c4c048f6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43939 } }
I20260929 22:54:21.952371 3526 raft_consensus.cc:388] T 00000000000000000000000000000000 P a39461715cd242589ae37187c4c048f6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 22:54:21.952406 3526 raft_consensus.cc:482] T 00000000000000000000000000000000 P a39461715cd242589ae37187c4c048f6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 22:54:21.952452 3526 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a39461715cd242589ae37187c4c048f6 [term 0 FOLLOWER]: Advancing to term 1
I20260929 22:54:21.953022 3526 raft_consensus.cc:504] T 00000000000000000000000000000000 P a39461715cd242589ae37187c4c048f6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a39461715cd242589ae37187c4c048f6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43939 } }
I20260929 22:54:21.953117 3526 leader_election.cc:302] T 00000000000000000000000000000000 P a39461715cd242589ae37187c4c048f6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a39461715cd242589ae37187c4c048f6; no voters:
I20260929 22:54:21.953212 14927 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhWOCLk/build/release/bin/kudu as pid 3458
I20260929 22:54:21.953289 3526 leader_election.cc:288] T 00000000000000000000000000000000 P a39461715cd242589ae37187c4c048f6 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 22:54:21.953326 14927 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790722213368738-14927-0/minicluster-data/master-0/wal/instance
I20260929 22:54:21.953346 3529 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a39461715cd242589ae37187c4c048f6 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 22:54:21.953471 3529 raft_consensus.cc:686] T 00000000000000000000000000000000 P a39461715cd242589ae37187c4c048f6 [term 1 LEADER]: Becoming Leader. State: Replica: a39461715cd242589ae37187c4c048f6, State: Running, Role: LEADER
I20260929 22:54:21.953548 14927 external_mini_cluster.cc:1428] Setting key 838c9bd99145d0785a558cce58d3eb25
I20260929 22:54:21.953610 3526 sys_catalog.cc:564] T 00000000000000000000000000000000 P a39461715cd242589ae37187c4c048f6 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 22:54:21.953610 3529 consensus_queue.cc:237] T 00000000000000000000000000000000 P a39461715cd242589ae37187c4c048f6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a39461715cd242589ae37187c4c048f6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43939 } }
I20260929 22:54:21.954165 3530 sys_catalog.cc:455] T 00000000000000000000000000000000 P a39461715cd242589ae37187c4c048f6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a39461715cd242589ae37187c4c048f6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a39461715cd242589ae37187c4c048f6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43939 } } }
I20260929 22:54:21.954250 3530 sys_catalog.cc:458] T 00000000000000000000000000000000 P a39461715cd242589ae37187c4c048f6 [sys.catalog]: This master's current role is: LEADER
I20260929 22:54:21.954449 3537 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 22:54:21.954454 3531 sys_catalog.cc:455] T 00000000000000000000000000000000 P a39461715cd242589ae37187c4c048f6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a39461715cd242589ae37187c4c048f6. Latest consensus state: current_term: 1 leader_uuid: "a39461715cd242589ae37187c4c048f6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a39461715cd242589ae37187c4c048f6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43939 } } }
I20260929 22:54:21.954624 3531 sys_catalog.cc:458] T 00000000000000000000000000000000 P a39461715cd242589ae37187c4c048f6 [sys.catalog]: This master's current role is: LEADER
I20260929 22:54:21.955046 3537 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 22:54:21.957574 3537 catalog_manager.cc:1347] Generated new cluster ID: 65cb7dbcafa64c2b874a6ed025fa59e6
I20260929 22:54:21.957636 3537 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 22:54:21.959684 3534 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:54:21.954010 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.71:37261 (local address 127.0.0.1:43939)
0929 22:54:21.954187 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0929 22:54:21.954197 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 22:54:21.954232 (+ 35us) server_negotiation.cc:407] Connection header received
0929 22:54:21.954357 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 22:54:21.954366 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 22:54:21.954426 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 22:54:21.954812 (+ 386us) server_negotiation.cc:226] Negotiated authn=SASL
0929 22:54:21.954909 (+ 97us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:54:21.956556 (+ 1647us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:54:21.957526 (+ 970us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:54:21.958117 (+ 591us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:54:21.958218 (+ 101us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 22:54:21.958971 (+ 753us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 22:54:21.958975 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 22:54:21.958975 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 22:54:21.959180 (+ 205us) server_negotiation.cc:922] calling sasl_server_start()
0929 22:54:21.959219 (+ 39us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 22:54:21.959241 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 22:54:21.959320 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0929 22:54:21.959441 (+ 121us) server_negotiation.cc:299] Negotiation successful
0929 22:54:21.959469 (+ 28us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":120,"thread_start_us":78,"threads_started":1}
I20260929 22:54:21.960704 14927 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 22:54:21.965715 3534 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:54:21.962106 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.71:44723 (local address 127.0.0.1:43939)
0929 22:54:21.962163 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0929 22:54:21.962165 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 22:54:21.962374 (+ 209us) server_negotiation.cc:407] Connection header received
0929 22:54:21.962434 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 22:54:21.962435 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 22:54:21.962447 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 22:54:21.962512 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0929 22:54:21.962628 (+ 116us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:54:21.963443 (+ 815us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:54:21.964425 (+ 982us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:54:21.964717 (+ 292us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:54:21.964789 (+ 72us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 22:54:21.965493 (+ 704us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 22:54:21.965494 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 22:54:21.965494 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 22:54:21.965499 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0929 22:54:21.965513 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 22:54:21.965518 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 22:54:21.965567 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0929 22:54:21.965633 (+ 66us) server_negotiation.cc:299] Negotiation successful
0929 22:54:21.965644 (+ 11us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260929 22:54:21.965871 3537 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 22:54:21.966450 3537 catalog_manager.cc:1504] Loading token signing keys...
I20260929 22:54:21.968726 3537 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a39461715cd242589ae37187c4c048f6: Generated new TSK 0
I20260929 22:54:21.968914 3537 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 22:54:21.972714 14927 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhWOCLk/build/release/bin/kudu with pid 3458
W20260929 22:54:21.976543 3443 connection.cc:535] client connection to 127.0.0.1:43939 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:43939): Connection reset by peer (error 104)
2026-09-29T22:54:21Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/7 (1210 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/8
Loading random data
Initializing database '/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790722213368738-14927-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 22:54:22 dist-test-slave-98mk krb5kdc[3557](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 22:54:22 dist-test-slave-98mk krb5kdc[3557](info): set up 2 sockets
Sep 29 22:54:22 dist-test-slave-98mk krb5kdc[3557](info): commencing operation
krb5kdc: starting...
W20260929 22:54:22.919993 14927 mini_kdc.cc:121] Time spent starting KDC: real 0.930s user 0.001s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790722213368738-14927-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790722213368738-14927-0/krb5kdc/test-user.keytab.
Sep 29 22:54:22 dist-test-slave-98mk krb5kdc[3557](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790722462, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T22:54:23Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T22:54:23Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790722213368738-14927-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790722213368738-14927-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260929 22:54:23.029836 14927 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhWOCLk/build/release/bin/kudu
/tmp/dist-test-taskhWOCLk/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790722213368738-14927-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790722213368738-14927-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790722213368738-14927-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790722213368738-14927-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45069
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790722213368738-14927-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:36783
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:45069
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790722213368738-14927-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790722213368738-14927-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790722213368738-14927-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 22:54:23.114396 3570 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 22:54:23.114636 3570 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 22:54:23.114679 3570 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 22:54:23.116079 3570 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 22:54:23.116137 3570 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 22:54:23.116155 3570 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 22:54:23.116173 3570 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 22:54:23.116187 3570 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 22:54:23.117834 3570 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:36783
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790722213368738-14927-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790722213368738-14927-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:45069
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790722213368738-14927-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45069
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790722213368738-14927-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3570
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790722213368738-14927-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 21:58:54 UTC on ccycloud.quasar-xguoti.root.comops.site
build id 4681390
I20260929 22:54:23.118095 3570 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 22:54:23.118312 3570 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 22:54:23.123267 3578 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 22:54:23.123206 3575 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 22:54:23.123203 3576 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 22:54:23.123349 3570 server_base.cc:1034] running on GCE node
I20260929 22:54:23.123850 3570 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 22:54:23.124233 3570 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 22:54:23.125439 3570 hybrid_clock.cc:648] HybridClock initialized: now 1790722463125406 us; error 51 us; skew 500 ppm
Sep 29 22:54:23 dist-test-slave-98mk krb5kdc[3557](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790722463, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 22:54:23.128238 3570 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260929 22:54:23.128795 3570 webserver.cc:460] Webserver started at http://127.0.0.1:37973/ using document root <none> and password file <none>
I20260929 22:54:23.128988 3570 fs_manager.cc:359] Metadata directory not provided
I20260929 22:54:23.129029 3570 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 22:54:23.129122 3570 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 22:54:23.130175 3570 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790722213368738-14927-0/minicluster-data/master-0/data/instance:
uuid: "1dc3b9f469f14f4fb77f23a55a54c7b4"
format_stamp: "Formatted at 2026-09-29 22:54:23 on dist-test-slave-98mk"
server_key: "9f8270c28cd0f1e73d275398a4bde510"
server_key_iv: "6891478d689418a6e64c5d96a185649c"
server_key_version: "encryptionkey@0"
I20260929 22:54:23.130486 3570 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790722213368738-14927-0/minicluster-data/master-0/wal/instance:
uuid: "1dc3b9f469f14f4fb77f23a55a54c7b4"
format_stamp: "Formatted at 2026-09-29 22:54:23 on dist-test-slave-98mk"
server_key: "9f8270c28cd0f1e73d275398a4bde510"
server_key_iv: "6891478d689418a6e64c5d96a185649c"
server_key_version: "encryptionkey@0"
I20260929 22:54:23.131826 3570 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260929 22:54:23.132607 3585 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 22:54:23.132876 3570 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260929 22:54:23.133008 3570 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790722213368738-14927-0/minicluster-data/master-0/data,/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790722213368738-14927-0/minicluster-data/master-0/wal
uuid: "1dc3b9f469f14f4fb77f23a55a54c7b4"
format_stamp: "Formatted at 2026-09-29 22:54:23 on dist-test-slave-98mk"
server_key: "9f8270c28cd0f1e73d275398a4bde510"
server_key_iv: "6891478d689418a6e64c5d96a185649c"
server_key_version: "encryptionkey@0"
I20260929 22:54:23.133100 3570 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790722213368738-14927-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790722213368738-14927-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790722213368738-14927-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 22:54:23.136952 3570 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 22:54:23.139077 3570 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 22:54:23.139195 3570 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 22:54:23.143025 3570 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:45069
I20260929 22:54:23.143074 3637 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:45069 every 8 connection(s)
I20260929 22:54:23.143392 3570 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790722213368738-14927-0/minicluster-data/master-0/data/info.pb
I20260929 22:54:23.144281 3638 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 22:54:23.145126 14927 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhWOCLk/build/release/bin/kudu as pid 3570
I20260929 22:54:23.145231 14927 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790722213368738-14927-0/minicluster-data/master-0/wal/instance
I20260929 22:54:23.145409 14927 external_mini_cluster.cc:1428] Setting key b5a85ae8a6fadbcd170d79b28e97cf3a
I20260929 22:54:23.146852 3638 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1dc3b9f469f14f4fb77f23a55a54c7b4: Bootstrap starting.
I20260929 22:54:23.147565 3638 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1dc3b9f469f14f4fb77f23a55a54c7b4: Neither blocks nor log segments found. Creating new log.
I20260929 22:54:23.147850 3638 log.cc:824] T 00000000000000000000000000000000 P 1dc3b9f469f14f4fb77f23a55a54c7b4: Log is configured to *not* fsync() on all Append() calls
I20260929 22:54:23.148380 3641 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:54:23.145917 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.71:49567 (local address 127.0.0.1:45069)
0929 22:54:23.146098 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0929 22:54:23.146107 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 22:54:23.146133 (+ 26us) server_negotiation.cc:407] Connection header received
0929 22:54:23.146660 (+ 527us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 22:54:23.146667 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 22:54:23.146694 (+ 27us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 22:54:23.146829 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0929 22:54:23.147697 (+ 868us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 22:54:23.147703 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 22:54:23.147703 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 22:54:23.147710 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0929 22:54:23.147773 (+ 63us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 22:54:23.147787 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 22:54:23.147882 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0929 22:54:23.147928 (+ 46us) server_negotiation.cc:299] Negotiation successful
0929 22:54:23.148172 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":134,"thread_start_us":83,"threads_started":1}
I20260929 22:54:23.148715 3638 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1dc3b9f469f14f4fb77f23a55a54c7b4: No bootstrap required, opened a new log
I20260929 22:54:23.149336 14927 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 22:54:23.150835 3638 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1dc3b9f469f14f4fb77f23a55a54c7b4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1dc3b9f469f14f4fb77f23a55a54c7b4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45069 } }
I20260929 22:54:23.150990 3638 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1dc3b9f469f14f4fb77f23a55a54c7b4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 22:54:23.151023 3638 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1dc3b9f469f14f4fb77f23a55a54c7b4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1dc3b9f469f14f4fb77f23a55a54c7b4, State: Initialized, Role: FOLLOWER
I20260929 22:54:23.151149 3638 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1dc3b9f469f14f4fb77f23a55a54c7b4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1dc3b9f469f14f4fb77f23a55a54c7b4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45069 } }
I20260929 22:54:23.151242 3638 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1dc3b9f469f14f4fb77f23a55a54c7b4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 22:54:23.151275 3638 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1dc3b9f469f14f4fb77f23a55a54c7b4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 22:54:23.151348 3638 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1dc3b9f469f14f4fb77f23a55a54c7b4 [term 0 FOLLOWER]: Advancing to term 1
I20260929 22:54:23.151734 3641 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:54:23.150652 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.71:37777 (local address 127.0.0.1:45069)
0929 22:54:23.150689 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0929 22:54:23.150691 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 22:54:23.150807 (+ 116us) server_negotiation.cc:407] Connection header received
0929 22:54:23.150866 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 22:54:23.150867 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 22:54:23.150872 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 22:54:23.150915 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0929 22:54:23.151540 (+ 625us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 22:54:23.151541 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 22:54:23.151541 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 22:54:23.151546 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0929 22:54:23.151561 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 22:54:23.151566 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 22:54:23.151614 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0929 22:54:23.151659 (+ 45us) server_negotiation.cc:299] Negotiation successful
0929 22:54:23.151672 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260929 22:54:23.152117 3638 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1dc3b9f469f14f4fb77f23a55a54c7b4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1dc3b9f469f14f4fb77f23a55a54c7b4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45069 } }
I20260929 22:54:23.152228 3638 leader_election.cc:302] T 00000000000000000000000000000000 P 1dc3b9f469f14f4fb77f23a55a54c7b4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1dc3b9f469f14f4fb77f23a55a54c7b4; no voters:
I20260929 22:54:23.152371 3638 leader_election.cc:288] T 00000000000000000000000000000000 P 1dc3b9f469f14f4fb77f23a55a54c7b4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 22:54:23.152417 3648 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1dc3b9f469f14f4fb77f23a55a54c7b4 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 22:54:23.152562 3648 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1dc3b9f469f14f4fb77f23a55a54c7b4 [term 1 LEADER]: Becoming Leader. State: Replica: 1dc3b9f469f14f4fb77f23a55a54c7b4, State: Running, Role: LEADER
I20260929 22:54:23.152604 3638 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1dc3b9f469f14f4fb77f23a55a54c7b4 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 22:54:23.152663 3648 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1dc3b9f469f14f4fb77f23a55a54c7b4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1dc3b9f469f14f4fb77f23a55a54c7b4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45069 } }
I20260929 22:54:23.153021 3649 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1dc3b9f469f14f4fb77f23a55a54c7b4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1dc3b9f469f14f4fb77f23a55a54c7b4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1dc3b9f469f14f4fb77f23a55a54c7b4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45069 } } }
I20260929 22:54:23.153033 3650 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1dc3b9f469f14f4fb77f23a55a54c7b4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1dc3b9f469f14f4fb77f23a55a54c7b4. Latest consensus state: current_term: 1 leader_uuid: "1dc3b9f469f14f4fb77f23a55a54c7b4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1dc3b9f469f14f4fb77f23a55a54c7b4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45069 } } }
I20260929 22:54:23.153155 3650 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1dc3b9f469f14f4fb77f23a55a54c7b4 [sys.catalog]: This master's current role is: LEADER
I20260929 22:54:23.153334 3649 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1dc3b9f469f14f4fb77f23a55a54c7b4 [sys.catalog]: This master's current role is: LEADER
I20260929 22:54:23.153607 3655 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 22:54:23.154064 3655 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 22:54:23.155797 3655 catalog_manager.cc:1347] Generated new cluster ID: 5d0e239578694ea0af2985bf69e15ee8
I20260929 22:54:23.155853 3655 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 22:54:23.178985 3655 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 22:54:23.179176 3655 catalog_manager.cc:1504] Loading token signing keys...
I20260929 22:54:23.188862 3655 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1dc3b9f469f14f4fb77f23a55a54c7b4: Generated new TSK 0
I20260929 22:54:23.189083 3655 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 22:54:23.195564 14927 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhWOCLk/build/release/bin/kudu with pid 3570
2026-09-29T22:54:23Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/8 (1222 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/9
Loading random data
Initializing database '/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790722213368738-14927-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 22:54:23 dist-test-slave-98mk krb5kdc[3669](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 22:54:23 dist-test-slave-98mk krb5kdc[3669](info): set up 2 sockets
Sep 29 22:54:23 dist-test-slave-98mk krb5kdc[3669](info): commencing operation
krb5kdc: starting...
W20260929 22:54:24.144078 14927 mini_kdc.cc:121] Time spent starting KDC: real 0.931s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790722213368738-14927-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790722213368738-14927-0/krb5kdc/test-user.keytab.
Sep 29 22:54:24 dist-test-slave-98mk krb5kdc[3669](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790722464, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T22:54:24Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T22:54:24Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790722213368738-14927-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790722213368738-14927-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260929 22:54:24.256058 14927 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhWOCLk/build/release/bin/kudu
/tmp/dist-test-taskhWOCLk/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790722213368738-14927-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790722213368738-14927-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790722213368738-14927-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790722213368738-14927-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35103
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790722213368738-14927-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:43301
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:35103
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790722213368738-14927-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790722213368738-14927-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790722213368738-14927-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 22:54:24.339032 3682 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 22:54:24.339272 3682 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 22:54:24.339298 3682 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 22:54:24.340548 3682 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 22:54:24.340610 3682 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 22:54:24.340623 3682 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 22:54:24.340636 3682 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 22:54:24.340647 3682 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 22:54:24.342182 3682 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:43301
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790722213368738-14927-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790722213368738-14927-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:35103
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790722213368738-14927-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35103
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790722213368738-14927-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3682
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790722213368738-14927-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 21:58:54 UTC on ccycloud.quasar-xguoti.root.comops.site
build id 4681390
I20260929 22:54:24.342420 3682 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 22:54:24.342613 3682 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 22:54:24.347707 3688 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 22:54:24.347716 3687 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 22:54:24.347780 3690 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 22:54:24.348140 3682 server_base.cc:1034] running on GCE node
I20260929 22:54:24.348405 3682 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 22:54:24.348708 3682 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 22:54:24.349854 3682 hybrid_clock.cc:648] HybridClock initialized: now 1790722464349843 us; error 33 us; skew 500 ppm
Sep 29 22:54:24 dist-test-slave-98mk krb5kdc[3669](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790722464, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 22:54:24.352573 3682 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260929 22:54:24.353027 3682 webserver.cc:460] Webserver started at http://127.0.0.1:41463/ using document root <none> and password file <none>
I20260929 22:54:24.353231 3682 fs_manager.cc:359] Metadata directory not provided
I20260929 22:54:24.353277 3682 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 22:54:24.353404 3682 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 22:54:24.354482 3682 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790722213368738-14927-0/minicluster-data/master-0/data/instance:
uuid: "06d4a54438584c4eb4ff038ee42135fc"
format_stamp: "Formatted at 2026-09-29 22:54:24 on dist-test-slave-98mk"
server_key: "69c98dbe724b7fee513f7b0c2bebfa45"
server_key_iv: "3639936b218698b8ec4548af5dfa3553"
server_key_version: "encryptionkey@0"
I20260929 22:54:24.354816 3682 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790722213368738-14927-0/minicluster-data/master-0/wal/instance:
uuid: "06d4a54438584c4eb4ff038ee42135fc"
format_stamp: "Formatted at 2026-09-29 22:54:24 on dist-test-slave-98mk"
server_key: "69c98dbe724b7fee513f7b0c2bebfa45"
server_key_iv: "3639936b218698b8ec4548af5dfa3553"
server_key_version: "encryptionkey@0"
I20260929 22:54:24.356304 3682 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260929 22:54:24.357278 3697 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 22:54:24.357476 3682 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260929 22:54:24.357577 3682 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790722213368738-14927-0/minicluster-data/master-0/data,/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790722213368738-14927-0/minicluster-data/master-0/wal
uuid: "06d4a54438584c4eb4ff038ee42135fc"
format_stamp: "Formatted at 2026-09-29 22:54:24 on dist-test-slave-98mk"
server_key: "69c98dbe724b7fee513f7b0c2bebfa45"
server_key_iv: "3639936b218698b8ec4548af5dfa3553"
server_key_version: "encryptionkey@0"
I20260929 22:54:24.357648 3682 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790722213368738-14927-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790722213368738-14927-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790722213368738-14927-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 22:54:24.368315 3682 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 22:54:24.370362 3682 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 22:54:24.370507 3682 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 22:54:24.374562 3682 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:35103
I20260929 22:54:24.374598 3749 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:35103 every 8 connection(s)
I20260929 22:54:24.374914 3682 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790722213368738-14927-0/minicluster-data/master-0/data/info.pb
I20260929 22:54:24.375809 3750 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 22:54:24.378085 3750 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 06d4a54438584c4eb4ff038ee42135fc: Bootstrap starting.
I20260929 22:54:24.378661 3750 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 06d4a54438584c4eb4ff038ee42135fc: Neither blocks nor log segments found. Creating new log.
I20260929 22:54:24.378922 3750 log.cc:824] T 00000000000000000000000000000000 P 06d4a54438584c4eb4ff038ee42135fc: Log is configured to *not* fsync() on all Append() calls
I20260929 22:54:24.379652 3750 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 06d4a54438584c4eb4ff038ee42135fc: No bootstrap required, opened a new log
I20260929 22:54:24.381218 3750 raft_consensus.cc:348] T 00000000000000000000000000000000 P 06d4a54438584c4eb4ff038ee42135fc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "06d4a54438584c4eb4ff038ee42135fc" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35103 } }
I20260929 22:54:24.381364 3750 raft_consensus.cc:374] T 00000000000000000000000000000000 P 06d4a54438584c4eb4ff038ee42135fc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 22:54:24.381392 3750 raft_consensus.cc:729] T 00000000000000000000000000000000 P 06d4a54438584c4eb4ff038ee42135fc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 06d4a54438584c4eb4ff038ee42135fc, State: Initialized, Role: FOLLOWER
I20260929 22:54:24.381541 3750 consensus_queue.cc:260] T 00000000000000000000000000000000 P 06d4a54438584c4eb4ff038ee42135fc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "06d4a54438584c4eb4ff038ee42135fc" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35103 } }
I20260929 22:54:24.381573 14927 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhWOCLk/build/release/bin/kudu as pid 3682
I20260929 22:54:24.381621 3750 raft_consensus.cc:388] T 00000000000000000000000000000000 P 06d4a54438584c4eb4ff038ee42135fc [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 22:54:24.381660 3750 raft_consensus.cc:482] T 00000000000000000000000000000000 P 06d4a54438584c4eb4ff038ee42135fc [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 22:54:24.381685 14927 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790722213368738-14927-0/minicluster-data/master-0/wal/instance
I20260929 22:54:24.381706 3750 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 06d4a54438584c4eb4ff038ee42135fc [term 0 FOLLOWER]: Advancing to term 1
I20260929 22:54:24.381865 14927 external_mini_cluster.cc:1428] Setting key 43e3a794586155c47b15512601c1d06f
I20260929 22:54:24.382370 3750 raft_consensus.cc:504] T 00000000000000000000000000000000 P 06d4a54438584c4eb4ff038ee42135fc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "06d4a54438584c4eb4ff038ee42135fc" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35103 } }
I20260929 22:54:24.382485 3750 leader_election.cc:302] T 00000000000000000000000000000000 P 06d4a54438584c4eb4ff038ee42135fc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 06d4a54438584c4eb4ff038ee42135fc; no voters:
I20260929 22:54:24.382652 3750 leader_election.cc:288] T 00000000000000000000000000000000 P 06d4a54438584c4eb4ff038ee42135fc [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 22:54:24.382694 3754 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 06d4a54438584c4eb4ff038ee42135fc [term 1 FOLLOWER]: Leader election won for term 1
I20260929 22:54:24.382853 3754 raft_consensus.cc:686] T 00000000000000000000000000000000 P 06d4a54438584c4eb4ff038ee42135fc [term 1 LEADER]: Becoming Leader. State: Replica: 06d4a54438584c4eb4ff038ee42135fc, State: Running, Role: LEADER
I20260929 22:54:24.382925 3750 sys_catalog.cc:564] T 00000000000000000000000000000000 P 06d4a54438584c4eb4ff038ee42135fc [sys.catalog]: configured and running, proceeding with master startup.
I20260929 22:54:24.382993 3754 consensus_queue.cc:237] T 00000000000000000000000000000000 P 06d4a54438584c4eb4ff038ee42135fc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "06d4a54438584c4eb4ff038ee42135fc" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35103 } }
I20260929 22:54:24.383317 3754 sys_catalog.cc:455] T 00000000000000000000000000000000 P 06d4a54438584c4eb4ff038ee42135fc [sys.catalog]: SysCatalogTable state changed. Reason: New leader 06d4a54438584c4eb4ff038ee42135fc. Latest consensus state: current_term: 1 leader_uuid: "06d4a54438584c4eb4ff038ee42135fc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "06d4a54438584c4eb4ff038ee42135fc" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35103 } } }
I20260929 22:54:24.383447 3754 sys_catalog.cc:458] T 00000000000000000000000000000000 P 06d4a54438584c4eb4ff038ee42135fc [sys.catalog]: This master's current role is: LEADER
I20260929 22:54:24.383698 3764 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 22:54:24.383812 3757 sys_catalog.cc:455] T 00000000000000000000000000000000 P 06d4a54438584c4eb4ff038ee42135fc [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "06d4a54438584c4eb4ff038ee42135fc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "06d4a54438584c4eb4ff038ee42135fc" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35103 } } }
I20260929 22:54:24.383935 3757 sys_catalog.cc:458] T 00000000000000000000000000000000 P 06d4a54438584c4eb4ff038ee42135fc [sys.catalog]: This master's current role is: LEADER
I20260929 22:54:24.384426 3764 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 22:54:24.388520 3764 catalog_manager.cc:1347] Generated new cluster ID: cf57b28674154e0aabaf9a28006f332e
I20260929 22:54:24.388583 3764 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 29 22:54:24 dist-test-slave-98mk krb5kdc[3669](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790722464, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260929 22:54:24.396538 3755 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:54:24.382528 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.71:38443 (local address 127.0.0.1:35103)
0929 22:54:24.383894 (+ 1366us) server_negotiation.cc:206] Beginning negotiation
0929 22:54:24.383904 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 22:54:24.383939 (+ 35us) server_negotiation.cc:407] Connection header received
0929 22:54:24.384071 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 22:54:24.384079 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 22:54:24.384148 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 22:54:24.384296 (+ 148us) server_negotiation.cc:226] Negotiated authn=SASL
0929 22:54:24.385033 (+ 737us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:54:24.386389 (+ 1356us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:54:24.387190 (+ 801us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:54:24.387859 (+ 669us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:54:24.387971 (+ 112us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 22:54:24.391584 (+ 3613us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 22:54:24.391592 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 22:54:24.391593 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 22:54:24.391614 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0929 22:54:24.394445 (+ 2831us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 22:54:24.395032 (+ 587us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 22:54:24.395036 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 22:54:24.395036 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 22:54:24.395162 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 22:54:24.395391 (+ 229us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 22:54:24.395393 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 22:54:24.395393 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 22:54:24.395812 (+ 419us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 22:54:24.395991 (+ 179us) server_negotiation.cc:1035] Waiting for connection context
0929 22:54:24.396119 (+ 128us) server_negotiation.cc:299] Negotiation successful
0929 22:54:24.396236 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1313,"thread_start_us":99,"threads_started":1}
I20260929 22:54:24.397852 14927 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 22:54:24.398252 3764 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 22:54:24.398751 3764 catalog_manager.cc:1504] Loading token signing keys...
I20260929 22:54:24.402992 3764 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 06d4a54438584c4eb4ff038ee42135fc: Generated new TSK 0
I20260929 22:54:24.403330 3764 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 22:54:24.406823 3755 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:54:24.399710 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.71:37971 (local address 127.0.0.1:35103)
0929 22:54:24.399755 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0929 22:54:24.399757 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 22:54:24.400061 (+ 304us) server_negotiation.cc:407] Connection header received
0929 22:54:24.400132 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 22:54:24.400133 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 22:54:24.400147 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 22:54:24.400201 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0929 22:54:24.400909 (+ 708us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:54:24.401723 (+ 814us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:54:24.402797 (+ 1074us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:54:24.403080 (+ 283us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:54:24.403144 (+ 64us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 22:54:24.404445 (+ 1301us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 22:54:24.404447 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 22:54:24.404448 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 22:54:24.404456 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0929 22:54:24.405741 (+ 1285us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 22:54:24.406102 (+ 361us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 22:54:24.406103 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 22:54:24.406104 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 22:54:24.406155 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 22:54:24.406344 (+ 189us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 22:54:24.406345 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 22:54:24.406346 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 22:54:24.406465 (+ 119us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 22:54:24.406562 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0929 22:54:24.406673 (+ 111us) server_negotiation.cc:299] Negotiation successful
0929 22:54:24.406732 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260929 22:54:24.409130 14927 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhWOCLk/build/release/bin/kudu with pid 3682
W20260929 22:54:24.414456 3667 connection.cc:535] client connection to 127.0.0.1:35103 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:35103): Connection reset by peer (error 104)
2026-09-29T22:54:24Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/9 (1215 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/10
Loading random data
Initializing database '/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790722213368738-14927-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 22:54:24 dist-test-slave-98mk krb5kdc[3781](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 22:54:24 dist-test-slave-98mk krb5kdc[3781](info): set up 2 sockets
Sep 29 22:54:24 dist-test-slave-98mk krb5kdc[3781](info): commencing operation
krb5kdc: starting...
W20260929 22:54:25.360075 14927 mini_kdc.cc:121] Time spent starting KDC: real 0.932s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790722213368738-14927-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790722213368738-14927-0/krb5kdc/test-user.keytab.
Sep 29 22:54:25 dist-test-slave-98mk krb5kdc[3781](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790722465, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T22:54:25Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T22:54:25Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790722213368738-14927-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790722213368738-14927-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260929 22:54:25.477447 14927 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhWOCLk/build/release/bin/kudu
/tmp/dist-test-taskhWOCLk/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790722213368738-14927-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790722213368738-14927-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790722213368738-14927-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790722213368738-14927-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37371
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790722213368738-14927-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:46661
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:37371
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790722213368738-14927-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790722213368738-14927-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790722213368738-14927-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 22:54:25.559387 3794 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 22:54:25.559661 3794 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 22:54:25.559689 3794 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 22:54:25.561257 3794 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 22:54:25.561316 3794 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 22:54:25.561331 3794 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 22:54:25.561342 3794 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 22:54:25.561352 3794 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 22:54:25.562989 3794 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:46661
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790722213368738-14927-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790722213368738-14927-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:37371
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790722213368738-14927-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37371
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790722213368738-14927-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3794
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790722213368738-14927-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 21:58:54 UTC on ccycloud.quasar-xguoti.root.comops.site
build id 4681390
I20260929 22:54:25.563231 3794 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 22:54:25.563473 3794 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 22:54:25.568053 3800 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 22:54:25.568046 3799 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 22:54:25.568046 3802 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 22:54:25.568255 3794 server_base.cc:1034] running on GCE node
I20260929 22:54:25.568487 3794 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 22:54:25.568749 3794 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 22:54:25.569890 3794 hybrid_clock.cc:648] HybridClock initialized: now 1790722465569876 us; error 28 us; skew 500 ppm
Sep 29 22:54:25 dist-test-slave-98mk krb5kdc[3781](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790722465, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 22:54:25.573037 3794 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260929 22:54:25.573488 3794 webserver.cc:460] Webserver started at http://127.0.0.1:46289/ using document root <none> and password file <none>
I20260929 22:54:25.573695 3794 fs_manager.cc:359] Metadata directory not provided
I20260929 22:54:25.573792 3794 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 22:54:25.573902 3794 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 22:54:25.574930 3794 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790722213368738-14927-0/minicluster-data/master-0/data/instance:
uuid: "f3e75a197b004acc95e6e52dccb996be"
format_stamp: "Formatted at 2026-09-29 22:54:25 on dist-test-slave-98mk"
server_key: "2f2ea251eb1a5a3d77fe3bf2462f08cc"
server_key_iv: "7f5fff43f2330d3801951b2f418e3d3b"
server_key_version: "encryptionkey@0"
I20260929 22:54:25.575225 3794 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790722213368738-14927-0/minicluster-data/master-0/wal/instance:
uuid: "f3e75a197b004acc95e6e52dccb996be"
format_stamp: "Formatted at 2026-09-29 22:54:25 on dist-test-slave-98mk"
server_key: "2f2ea251eb1a5a3d77fe3bf2462f08cc"
server_key_iv: "7f5fff43f2330d3801951b2f418e3d3b"
server_key_version: "encryptionkey@0"
I20260929 22:54:25.576529 3794 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260929 22:54:25.577234 3809 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 22:54:25.577387 3794 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260929 22:54:25.577474 3794 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790722213368738-14927-0/minicluster-data/master-0/data,/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790722213368738-14927-0/minicluster-data/master-0/wal
uuid: "f3e75a197b004acc95e6e52dccb996be"
format_stamp: "Formatted at 2026-09-29 22:54:25 on dist-test-slave-98mk"
server_key: "2f2ea251eb1a5a3d77fe3bf2462f08cc"
server_key_iv: "7f5fff43f2330d3801951b2f418e3d3b"
server_key_version: "encryptionkey@0"
I20260929 22:54:25.577561 3794 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790722213368738-14927-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790722213368738-14927-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790722213368738-14927-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 22:54:25.606122 3794 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 22:54:25.608064 3794 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 22:54:25.608182 3794 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 22:54:25.611999 3794 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:37371
I20260929 22:54:25.612047 3861 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:37371 every 8 connection(s)
I20260929 22:54:25.612371 3794 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790722213368738-14927-0/minicluster-data/master-0/data/info.pb
I20260929 22:54:25.612526 14927 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhWOCLk/build/release/bin/kudu as pid 3794
I20260929 22:54:25.612648 14927 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790722213368738-14927-0/minicluster-data/master-0/wal/instance
I20260929 22:54:25.612852 14927 external_mini_cluster.cc:1428] Setting key 0504887bc13070175dd411d86c0522e6
I20260929 22:54:25.613327 3862 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 22:54:25.616225 3862 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f3e75a197b004acc95e6e52dccb996be: Bootstrap starting.
I20260929 22:54:25.616814 3862 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f3e75a197b004acc95e6e52dccb996be: Neither blocks nor log segments found. Creating new log.
I20260929 22:54:25.617098 3862 log.cc:824] T 00000000000000000000000000000000 P f3e75a197b004acc95e6e52dccb996be: Log is configured to *not* fsync() on all Append() calls
I20260929 22:54:25.617762 3862 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f3e75a197b004acc95e6e52dccb996be: No bootstrap required, opened a new log
I20260929 22:54:25.618048 3865 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:54:25.613441 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.71:46779 (local address 127.0.0.1:37371)
0929 22:54:25.613614 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0929 22:54:25.613622 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 22:54:25.613647 (+ 25us) server_negotiation.cc:407] Connection header received
0929 22:54:25.614087 (+ 440us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 22:54:25.614094 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 22:54:25.614142 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 22:54:25.614258 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0929 22:54:25.614491 (+ 233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:54:25.615371 (+ 880us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:54:25.616166 (+ 795us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:54:25.616625 (+ 459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:54:25.616722 (+ 97us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 22:54:25.617441 (+ 719us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 22:54:25.617445 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 22:54:25.617445 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 22:54:25.617618 (+ 173us) server_negotiation.cc:922] calling sasl_server_start()
0929 22:54:25.617651 (+ 33us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 22:54:25.617661 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 22:54:25.617730 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0929 22:54:25.617823 (+ 93us) server_negotiation.cc:299] Negotiation successful
0929 22:54:25.617853 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":119,"thread_start_us":70,"threads_started":1}
I20260929 22:54:25.618957 14927 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 22:54:25.619946 3862 raft_consensus.cc:348] T 00000000000000000000000000000000 P f3e75a197b004acc95e6e52dccb996be [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f3e75a197b004acc95e6e52dccb996be" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37371 } }
I20260929 22:54:25.620139 3862 raft_consensus.cc:374] T 00000000000000000000000000000000 P f3e75a197b004acc95e6e52dccb996be [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 22:54:25.620189 3862 raft_consensus.cc:729] T 00000000000000000000000000000000 P f3e75a197b004acc95e6e52dccb996be [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f3e75a197b004acc95e6e52dccb996be, State: Initialized, Role: FOLLOWER
I20260929 22:54:25.620316 3862 consensus_queue.cc:260] T 00000000000000000000000000000000 P f3e75a197b004acc95e6e52dccb996be [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f3e75a197b004acc95e6e52dccb996be" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37371 } }
I20260929 22:54:25.620396 3862 raft_consensus.cc:388] T 00000000000000000000000000000000 P f3e75a197b004acc95e6e52dccb996be [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 22:54:25.620421 3862 raft_consensus.cc:482] T 00000000000000000000000000000000 P f3e75a197b004acc95e6e52dccb996be [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 22:54:25.620471 3862 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f3e75a197b004acc95e6e52dccb996be [term 0 FOLLOWER]: Advancing to term 1
I20260929 22:54:25.621196 3862 raft_consensus.cc:504] T 00000000000000000000000000000000 P f3e75a197b004acc95e6e52dccb996be [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f3e75a197b004acc95e6e52dccb996be" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37371 } }
I20260929 22:54:25.621295 3862 leader_election.cc:302] T 00000000000000000000000000000000 P f3e75a197b004acc95e6e52dccb996be [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f3e75a197b004acc95e6e52dccb996be; no voters:
I20260929 22:54:25.621449 3862 leader_election.cc:288] T 00000000000000000000000000000000 P f3e75a197b004acc95e6e52dccb996be [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 22:54:25.621492 3872 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f3e75a197b004acc95e6e52dccb996be [term 1 FOLLOWER]: Leader election won for term 1
I20260929 22:54:25.621650 3872 raft_consensus.cc:686] T 00000000000000000000000000000000 P f3e75a197b004acc95e6e52dccb996be [term 1 LEADER]: Becoming Leader. State: Replica: f3e75a197b004acc95e6e52dccb996be, State: Running, Role: LEADER
I20260929 22:54:25.621721 3862 sys_catalog.cc:564] T 00000000000000000000000000000000 P f3e75a197b004acc95e6e52dccb996be [sys.catalog]: configured and running, proceeding with master startup.
I20260929 22:54:25.621809 3872 consensus_queue.cc:237] T 00000000000000000000000000000000 P f3e75a197b004acc95e6e52dccb996be [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f3e75a197b004acc95e6e52dccb996be" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37371 } }
I20260929 22:54:25.622174 3872 sys_catalog.cc:455] T 00000000000000000000000000000000 P f3e75a197b004acc95e6e52dccb996be [sys.catalog]: SysCatalogTable state changed. Reason: New leader f3e75a197b004acc95e6e52dccb996be. Latest consensus state: current_term: 1 leader_uuid: "f3e75a197b004acc95e6e52dccb996be" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f3e75a197b004acc95e6e52dccb996be" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37371 } } }
I20260929 22:54:25.622184 3874 sys_catalog.cc:455] T 00000000000000000000000000000000 P f3e75a197b004acc95e6e52dccb996be [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f3e75a197b004acc95e6e52dccb996be" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f3e75a197b004acc95e6e52dccb996be" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37371 } } }
I20260929 22:54:25.622344 3874 sys_catalog.cc:458] T 00000000000000000000000000000000 P f3e75a197b004acc95e6e52dccb996be [sys.catalog]: This master's current role is: LEADER
I20260929 22:54:25.622529 3872 sys_catalog.cc:458] T 00000000000000000000000000000000 P f3e75a197b004acc95e6e52dccb996be [sys.catalog]: This master's current role is: LEADER
I20260929 22:54:25.622694 3881 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 22:54:25.623600 3881 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 22:54:25.623899 3865 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:54:25.620475 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.71:40211 (local address 127.0.0.1:37371)
0929 22:54:25.620519 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0929 22:54:25.620521 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 22:54:25.620711 (+ 190us) server_negotiation.cc:407] Connection header received
0929 22:54:25.620774 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 22:54:25.620774 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 22:54:25.620784 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 22:54:25.620827 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0929 22:54:25.620964 (+ 137us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:54:25.621484 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:54:25.622408 (+ 924us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:54:25.622689 (+ 281us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:54:25.622750 (+ 61us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 22:54:25.623503 (+ 753us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 22:54:25.623505 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 22:54:25.623505 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 22:54:25.623525 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0929 22:54:25.623541 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 22:54:25.623546 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 22:54:25.623597 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0929 22:54:25.623811 (+ 214us) server_negotiation.cc:299] Negotiation successful
0929 22:54:25.623825 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260929 22:54:25.625156 3881 catalog_manager.cc:1347] Generated new cluster ID: 426f4550417249a8acec62802b1d7f06
I20260929 22:54:25.625200 3881 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 22:54:25.635849 3881 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 22:54:25.636459 3881 catalog_manager.cc:1504] Loading token signing keys...
I20260929 22:54:25.639767 3881 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f3e75a197b004acc95e6e52dccb996be: Generated new TSK 0
I20260929 22:54:25.639997 3881 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 22:54:25.644078 14927 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhWOCLk/build/release/bin/kudu with pid 3794
W20260929 22:54:25.648051 3779 connection.cc:535] client connection to 127.0.0.1:37371 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:37371): Connection reset by peer (error 104)
2026-09-29T22:54:25Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/10 (1233 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/11
Loading random data
Initializing database '/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790722213368738-14927-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 22:54:25 dist-test-slave-98mk krb5kdc[3893](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 22:54:25 dist-test-slave-98mk krb5kdc[3893](info): set up 2 sockets
Sep 29 22:54:25 dist-test-slave-98mk krb5kdc[3893](info): commencing operation
krb5kdc: starting...
W20260929 22:54:26.588043 14927 mini_kdc.cc:121] Time spent starting KDC: real 0.926s user 0.002s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790722213368738-14927-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790722213368738-14927-0/krb5kdc/test-user.keytab.
Sep 29 22:54:26 dist-test-slave-98mk krb5kdc[3893](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790722466, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T22:54:26Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T22:54:26Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790722213368738-14927-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790722213368738-14927-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260929 22:54:26.702378 14927 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhWOCLk/build/release/bin/kudu
/tmp/dist-test-taskhWOCLk/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790722213368738-14927-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790722213368738-14927-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790722213368738-14927-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790722213368738-14927-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40379
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790722213368738-14927-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:43753
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:40379
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790722213368738-14927-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790722213368738-14927-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790722213368738-14927-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 22:54:26.782713 3906 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 22:54:26.782997 3906 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 22:54:26.783034 3906 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 22:54:26.784477 3906 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 22:54:26.784525 3906 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 22:54:26.784539 3906 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 22:54:26.784551 3906 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 22:54:26.784562 3906 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 22:54:26.786068 3906 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:43753
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790722213368738-14927-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790722213368738-14927-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:40379
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790722213368738-14927-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40379
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790722213368738-14927-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3906
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790722213368738-14927-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 21:58:54 UTC on ccycloud.quasar-xguoti.root.comops.site
build id 4681390
I20260929 22:54:26.786276 3906 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 22:54:26.786458 3906 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 22:54:26.790946 3914 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 22:54:26.790940 3911 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 22:54:26.790940 3912 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 22:54:26.790999 3906 server_base.cc:1034] running on GCE node
I20260929 22:54:26.791381 3906 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 22:54:26.791702 3906 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 22:54:26.792843 3906 hybrid_clock.cc:648] HybridClock initialized: now 1790722466792826 us; error 33 us; skew 500 ppm
Sep 29 22:54:26 dist-test-slave-98mk krb5kdc[3893](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790722466, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 22:54:26.799971 3906 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260929 22:54:26.800454 3906 webserver.cc:460] Webserver started at http://127.0.0.1:34939/ using document root <none> and password file <none>
I20260929 22:54:26.800671 3906 fs_manager.cc:359] Metadata directory not provided
I20260929 22:54:26.800721 3906 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 22:54:26.800848 3906 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 22:54:26.801928 3906 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790722213368738-14927-0/minicluster-data/master-0/data/instance:
uuid: "8a31003ac3b641b9b261df8a090fa487"
format_stamp: "Formatted at 2026-09-29 22:54:26 on dist-test-slave-98mk"
server_key: "76e64fe8341b12532a2f9ab0ea46241c"
server_key_iv: "feba52568f2966224883a70b75138cfa"
server_key_version: "encryptionkey@0"
I20260929 22:54:26.802258 3906 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790722213368738-14927-0/minicluster-data/master-0/wal/instance:
uuid: "8a31003ac3b641b9b261df8a090fa487"
format_stamp: "Formatted at 2026-09-29 22:54:26 on dist-test-slave-98mk"
server_key: "76e64fe8341b12532a2f9ab0ea46241c"
server_key_iv: "feba52568f2966224883a70b75138cfa"
server_key_version: "encryptionkey@0"
I20260929 22:54:26.803655 3906 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.001s
I20260929 22:54:26.804440 3921 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 22:54:26.804615 3906 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260929 22:54:26.804697 3906 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790722213368738-14927-0/minicluster-data/master-0/data,/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790722213368738-14927-0/minicluster-data/master-0/wal
uuid: "8a31003ac3b641b9b261df8a090fa487"
format_stamp: "Formatted at 2026-09-29 22:54:26 on dist-test-slave-98mk"
server_key: "76e64fe8341b12532a2f9ab0ea46241c"
server_key_iv: "feba52568f2966224883a70b75138cfa"
server_key_version: "encryptionkey@0"
I20260929 22:54:26.804760 3906 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790722213368738-14927-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790722213368738-14927-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790722213368738-14927-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 22:54:26.808252 3906 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 22:54:26.810211 3906 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 22:54:26.810348 3906 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 22:54:26.814257 3906 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:40379
I20260929 22:54:26.814294 3973 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:40379 every 8 connection(s)
I20260929 22:54:26.814605 3906 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790722213368738-14927-0/minicluster-data/master-0/data/info.pb
I20260929 22:54:26.815515 3974 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 22:54:26.817317 14927 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhWOCLk/build/release/bin/kudu as pid 3906
I20260929 22:54:26.817435 14927 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790722213368738-14927-0/minicluster-data/master-0/wal/instance
I20260929 22:54:26.817598 14927 external_mini_cluster.cc:1428] Setting key 5ccc65c21e3138790005b09ac06c0e36
I20260929 22:54:26.817812 3974 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8a31003ac3b641b9b261df8a090fa487: Bootstrap starting.
I20260929 22:54:26.818466 3974 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8a31003ac3b641b9b261df8a090fa487: Neither blocks nor log segments found. Creating new log.
I20260929 22:54:26.818765 3974 log.cc:824] T 00000000000000000000000000000000 P 8a31003ac3b641b9b261df8a090fa487: Log is configured to *not* fsync() on all Append() calls
I20260929 22:54:26.819483 3974 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8a31003ac3b641b9b261df8a090fa487: No bootstrap required, opened a new log
I20260929 22:54:26.820217 3977 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:54:26.818207 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.71:42267 (local address 127.0.0.1:40379)
0929 22:54:26.818381 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0929 22:54:26.818389 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 22:54:26.818421 (+ 32us) server_negotiation.cc:407] Connection header received
0929 22:54:26.819011 (+ 590us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 22:54:26.819020 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 22:54:26.819047 (+ 27us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 22:54:26.819173 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0929 22:54:26.819848 (+ 675us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 22:54:26.819851 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 22:54:26.819852 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 22:54:26.819856 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0929 22:54:26.819905 (+ 49us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 22:54:26.819914 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 22:54:26.819987 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0929 22:54:26.820020 (+ 33us) server_negotiation.cc:299] Negotiation successful
0929 22:54:26.820060 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":115,"thread_start_us":69,"threads_started":1}
I20260929 22:54:26.821087 14927 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 22:54:26.821482 3974 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8a31003ac3b641b9b261df8a090fa487 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8a31003ac3b641b9b261df8a090fa487" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40379 } }
I20260929 22:54:26.821614 3974 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8a31003ac3b641b9b261df8a090fa487 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 22:54:26.821645 3974 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8a31003ac3b641b9b261df8a090fa487 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a31003ac3b641b9b261df8a090fa487, State: Initialized, Role: FOLLOWER
I20260929 22:54:26.821758 3974 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8a31003ac3b641b9b261df8a090fa487 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8a31003ac3b641b9b261df8a090fa487" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40379 } }
I20260929 22:54:26.821839 3974 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8a31003ac3b641b9b261df8a090fa487 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 22:54:26.821879 3974 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8a31003ac3b641b9b261df8a090fa487 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 22:54:26.821925 3974 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8a31003ac3b641b9b261df8a090fa487 [term 0 FOLLOWER]: Advancing to term 1
I20260929 22:54:26.822520 3974 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8a31003ac3b641b9b261df8a090fa487 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8a31003ac3b641b9b261df8a090fa487" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40379 } }
I20260929 22:54:26.822623 3974 leader_election.cc:302] T 00000000000000000000000000000000 P 8a31003ac3b641b9b261df8a090fa487 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8a31003ac3b641b9b261df8a090fa487; no voters:
I20260929 22:54:26.822782 3974 leader_election.cc:288] T 00000000000000000000000000000000 P 8a31003ac3b641b9b261df8a090fa487 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 22:54:26.822824 3984 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8a31003ac3b641b9b261df8a090fa487 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 22:54:26.822954 3984 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8a31003ac3b641b9b261df8a090fa487 [term 1 LEADER]: Becoming Leader. State: Replica: 8a31003ac3b641b9b261df8a090fa487, State: Running, Role: LEADER
I20260929 22:54:26.823024 3974 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8a31003ac3b641b9b261df8a090fa487 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 22:54:26.823041 3984 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8a31003ac3b641b9b261df8a090fa487 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8a31003ac3b641b9b261df8a090fa487" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40379 } }
I20260929 22:54:26.823405 3977 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:54:26.822386 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.71:36735 (local address 127.0.0.1:40379)
0929 22:54:26.822420 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0929 22:54:26.822421 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0929 22:54:26.822493 (+ 72us) server_negotiation.cc:407] Connection header received
0929 22:54:26.822564 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 22:54:26.822565 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 22:54:26.822568 (+ 3us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 22:54:26.822615 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0929 22:54:26.823181 (+ 566us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 22:54:26.823182 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 22:54:26.823182 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 22:54:26.823187 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0929 22:54:26.823200 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 22:54:26.823203 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 22:54:26.823231 (+ 28us) server_negotiation.cc:1035] Waiting for connection context
0929 22:54:26.823324 (+ 93us) server_negotiation.cc:299] Negotiation successful
0929 22:54:26.823337 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260929 22:54:26.824003 3986 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8a31003ac3b641b9b261df8a090fa487 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8a31003ac3b641b9b261df8a090fa487. Latest consensus state: current_term: 1 leader_uuid: "8a31003ac3b641b9b261df8a090fa487" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8a31003ac3b641b9b261df8a090fa487" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40379 } } }
I20260929 22:54:26.824152 3986 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8a31003ac3b641b9b261df8a090fa487 [sys.catalog]: This master's current role is: LEADER
I20260929 22:54:26.824203 3985 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8a31003ac3b641b9b261df8a090fa487 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8a31003ac3b641b9b261df8a090fa487" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8a31003ac3b641b9b261df8a090fa487" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40379 } } }
I20260929 22:54:26.824285 3985 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8a31003ac3b641b9b261df8a090fa487 [sys.catalog]: This master's current role is: LEADER
W20260929 22:54:26.824702 4000 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 8a31003ac3b641b9b261df8a090fa487: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 22:54:26.824741 4000 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260929 22:54:26.824806 3999 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 22:54:26.824977 3999 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 22:54:26.826350 3999 catalog_manager.cc:1347] Generated new cluster ID: 58d10da99deb41ea8919b3c1c811b0dc
I20260929 22:54:26.826403 3999 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 22:54:26.852077 3999 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 22:54:26.852274 3999 catalog_manager.cc:1504] Loading token signing keys...
I20260929 22:54:26.855850 3999 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8a31003ac3b641b9b261df8a090fa487: Generated new TSK 0
I20260929 22:54:26.856031 3999 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 22:54:26.857692 14927 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhWOCLk/build/release/bin/kudu with pid 3906
2026-09-29T22:54:26Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/11 (1213 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/12
Loading random data
Initializing database '/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790722213368738-14927-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 22:54:26 dist-test-slave-98mk krb5kdc[4005](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 22:54:26 dist-test-slave-98mk krb5kdc[4005](info): set up 2 sockets
Sep 29 22:54:26 dist-test-slave-98mk krb5kdc[4005](info): commencing operation
krb5kdc: starting...
W20260929 22:54:27.812079 14927 mini_kdc.cc:121] Time spent starting KDC: real 0.937s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790722213368738-14927-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790722213368738-14927-0/krb5kdc/test-user.keytab.
Sep 29 22:54:27 dist-test-slave-98mk krb5kdc[4005](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790722467, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T22:54:27Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T22:54:27Z Disabled control of system clock
WARNING: no policy specified for kudu/127.14.147.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.147.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.147.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790722213368738-14927-0/krb5kdc/kudu_127.14.147.254.keytab.
Entry for principal kudu/127.14.147.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790722213368738-14927-0/krb5kdc/kudu_127.14.147.254.keytab.
I20260929 22:54:27.924259 14927 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhWOCLk/build/release/bin/kudu
/tmp/dist-test-taskhWOCLk/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790722213368738-14927-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790722213368738-14927-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790722213368738-14927-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790722213368738-14927-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.147.254:42281
--webserver_interface=127.14.147.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790722213368738-14927-0/krb5kdc/kudu_127.14.147.254.keytab
--principal=kudu/127.14.147.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.147.212:43881
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.147.254:42281
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790722213368738-14927-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790722213368738-14927-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790722213368738-14927-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 22:54:28.006099 4018 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 22:54:28.006330 4018 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 22:54:28.006356 4018 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 22:54:28.008832 4018 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 22:54:28.008932 4018 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 22:54:28.008950 4018 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 22:54:28.008965 4018 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 22:54:28.008980 4018 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 22:54:28.011725 4018 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.147.212:43881
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790722213368738-14927-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790722213368738-14927-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.14.147.254:42281
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790722213368738-14927-0/krb5kdc/kudu_127.14.147.254.keytab
--principal=kudu/127.14.147.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.147.254:42281
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790722213368738-14927-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.14.147.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4018
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790722213368738-14927-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 21:58:54 UTC on ccycloud.quasar-xguoti.root.comops.site
build id 4681390
I20260929 22:54:28.012017 4018 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 22:54:28.012217 4018 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 22:54:28.017123 4026 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 22:54:28.017168 4024 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 22:54:28.017187 4023 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 22:54:28.017308 4018 server_base.cc:1034] running on GCE node
I20260929 22:54:28.017669 4018 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 22:54:28.017966 4018 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 22:54:28.019129 4018 hybrid_clock.cc:648] HybridClock initialized: now 1790722468019109 us; error 32 us; skew 500 ppm
Sep 29 22:54:28 dist-test-slave-98mk krb5kdc[4005](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790722468, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.147.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 22:54:28.021543 4018 init.cc:377] Logged in from keytab as kudu/127.14.147.254@KRBTEST.COM (short username kudu)
I20260929 22:54:28.022024 4018 webserver.cc:460] Webserver started at http://127.14.147.254:44997/ using document root <none> and password file <none>
I20260929 22:54:28.022214 4018 fs_manager.cc:359] Metadata directory not provided
I20260929 22:54:28.022253 4018 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 22:54:28.022361 4018 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 22:54:28.023464 4018 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790722213368738-14927-0/minicluster-data/master-0/data/instance:
uuid: "d1b8ae21083e4936aa9d1faf2df93d8b"
format_stamp: "Formatted at 2026-09-29 22:54:28 on dist-test-slave-98mk"
server_key: "cb6a2f3bd7c508f1ae4d05f9080d69eb"
server_key_iv: "c7a60025dd8ee93b1f04c6bea81e3cc6"
server_key_version: "encryptionkey@0"
I20260929 22:54:28.023784 4018 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790722213368738-14927-0/minicluster-data/master-0/wal/instance:
uuid: "d1b8ae21083e4936aa9d1faf2df93d8b"
format_stamp: "Formatted at 2026-09-29 22:54:28 on dist-test-slave-98mk"
server_key: "cb6a2f3bd7c508f1ae4d05f9080d69eb"
server_key_iv: "c7a60025dd8ee93b1f04c6bea81e3cc6"
server_key_version: "encryptionkey@0"
I20260929 22:54:28.025044 4018 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260929 22:54:28.025785 4033 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 22:54:28.025954 4018 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260929 22:54:28.026036 4018 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790722213368738-14927-0/minicluster-data/master-0/data,/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790722213368738-14927-0/minicluster-data/master-0/wal
uuid: "d1b8ae21083e4936aa9d1faf2df93d8b"
format_stamp: "Formatted at 2026-09-29 22:54:28 on dist-test-slave-98mk"
server_key: "cb6a2f3bd7c508f1ae4d05f9080d69eb"
server_key_iv: "c7a60025dd8ee93b1f04c6bea81e3cc6"
server_key_version: "encryptionkey@0"
I20260929 22:54:28.026100 4018 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790722213368738-14927-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790722213368738-14927-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790722213368738-14927-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 22:54:28.049973 4018 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 22:54:28.051836 4018 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 22:54:28.051949 4018 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 22:54:28.055773 4018 rpc_server.cc:307] RPC server started. Bound to: 127.14.147.254:42281
I20260929 22:54:28.055904 4085 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.147.254:42281 every 8 connection(s)
I20260929 22:54:28.056198 4018 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790722213368738-14927-0/minicluster-data/master-0/data/info.pb
I20260929 22:54:28.057008 4086 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 22:54:28.059221 4086 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d1b8ae21083e4936aa9d1faf2df93d8b: Bootstrap starting.
I20260929 22:54:28.059517 14927 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhWOCLk/build/release/bin/kudu as pid 4018
I20260929 22:54:28.059625 14927 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790722213368738-14927-0/minicluster-data/master-0/wal/instance
I20260929 22:54:28.059798 14927 external_mini_cluster.cc:1428] Setting key e1400511fdef22db84672fd3222743c1
I20260929 22:54:28.059896 4086 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d1b8ae21083e4936aa9d1faf2df93d8b: Neither blocks nor log segments found. Creating new log.
I20260929 22:54:28.060158 4086 log.cc:824] T 00000000000000000000000000000000 P d1b8ae21083e4936aa9d1faf2df93d8b: Log is configured to *not* fsync() on all Append() calls
I20260929 22:54:28.060922 4086 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d1b8ae21083e4936aa9d1faf2df93d8b: No bootstrap required, opened a new log
I20260929 22:54:28.063153 4086 raft_consensus.cc:348] T 00000000000000000000000000000000 P d1b8ae21083e4936aa9d1faf2df93d8b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d1b8ae21083e4936aa9d1faf2df93d8b" member_type: VOTER last_known_addr { host: "127.14.147.254" port: 42281 } }
I20260929 22:54:28.063329 4086 raft_consensus.cc:374] T 00000000000000000000000000000000 P d1b8ae21083e4936aa9d1faf2df93d8b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 22:54:28.063362 4086 raft_consensus.cc:729] T 00000000000000000000000000000000 P d1b8ae21083e4936aa9d1faf2df93d8b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d1b8ae21083e4936aa9d1faf2df93d8b, State: Initialized, Role: FOLLOWER
I20260929 22:54:28.063517 4086 consensus_queue.cc:260] T 00000000000000000000000000000000 P d1b8ae21083e4936aa9d1faf2df93d8b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d1b8ae21083e4936aa9d1faf2df93d8b" member_type: VOTER last_known_addr { host: "127.14.147.254" port: 42281 } }
I20260929 22:54:28.063616 4086 raft_consensus.cc:388] T 00000000000000000000000000000000 P d1b8ae21083e4936aa9d1faf2df93d8b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 22:54:28.063648 4086 raft_consensus.cc:482] T 00000000000000000000000000000000 P d1b8ae21083e4936aa9d1faf2df93d8b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 22:54:28.063699 4086 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d1b8ae21083e4936aa9d1faf2df93d8b [term 0 FOLLOWER]: Advancing to term 1
I20260929 22:54:28.064420 4086 raft_consensus.cc:504] T 00000000000000000000000000000000 P d1b8ae21083e4936aa9d1faf2df93d8b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d1b8ae21083e4936aa9d1faf2df93d8b" member_type: VOTER last_known_addr { host: "127.14.147.254" port: 42281 } }
I20260929 22:54:28.064527 4086 leader_election.cc:302] T 00000000000000000000000000000000 P d1b8ae21083e4936aa9d1faf2df93d8b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d1b8ae21083e4936aa9d1faf2df93d8b; no voters:
I20260929 22:54:28.064670 4086 leader_election.cc:288] T 00000000000000000000000000000000 P d1b8ae21083e4936aa9d1faf2df93d8b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 22:54:28.064715 4091 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d1b8ae21083e4936aa9d1faf2df93d8b [term 1 FOLLOWER]: Leader election won for term 1
I20260929 22:54:28.064920 4091 raft_consensus.cc:686] T 00000000000000000000000000000000 P d1b8ae21083e4936aa9d1faf2df93d8b [term 1 LEADER]: Becoming Leader. State: Replica: d1b8ae21083e4936aa9d1faf2df93d8b, State: Running, Role: LEADER
I20260929 22:54:28.064927 4086 sys_catalog.cc:564] T 00000000000000000000000000000000 P d1b8ae21083e4936aa9d1faf2df93d8b [sys.catalog]: configured and running, proceeding with master startup.
I20260929 22:54:28.065104 4091 consensus_queue.cc:237] T 00000000000000000000000000000000 P d1b8ae21083e4936aa9d1faf2df93d8b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d1b8ae21083e4936aa9d1faf2df93d8b" member_type: VOTER last_known_addr { host: "127.14.147.254" port: 42281 } }
I20260929 22:54:28.065516 4092 sys_catalog.cc:455] T 00000000000000000000000000000000 P d1b8ae21083e4936aa9d1faf2df93d8b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d1b8ae21083e4936aa9d1faf2df93d8b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d1b8ae21083e4936aa9d1faf2df93d8b" member_type: VOTER last_known_addr { host: "127.14.147.254" port: 42281 } } }
I20260929 22:54:28.065596 4092 sys_catalog.cc:458] T 00000000000000000000000000000000 P d1b8ae21083e4936aa9d1faf2df93d8b [sys.catalog]: This master's current role is: LEADER
I20260929 22:54:28.065850 4093 sys_catalog.cc:455] T 00000000000000000000000000000000 P d1b8ae21083e4936aa9d1faf2df93d8b [sys.catalog]: SysCatalogTable state changed. Reason: New leader d1b8ae21083e4936aa9d1faf2df93d8b. Latest consensus state: current_term: 1 leader_uuid: "d1b8ae21083e4936aa9d1faf2df93d8b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d1b8ae21083e4936aa9d1faf2df93d8b" member_type: VOTER last_known_addr { host: "127.14.147.254" port: 42281 } } }
I20260929 22:54:28.066169 4093 sys_catalog.cc:458] T 00000000000000000000000000000000 P d1b8ae21083e4936aa9d1faf2df93d8b [sys.catalog]: This master's current role is: LEADER
W20260929 22:54:28.067214 4107 catalog_manager.cc:1558] T 00000000000000000000000000000000 P d1b8ae21083e4936aa9d1faf2df93d8b: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 22:54:28.067556 4107 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
Sep 29 22:54:28 dist-test-slave-98mk krb5kdc[4005](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790722467, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.147.254@KRBTEST.COM
I20260929 22:54:28.067632 4096 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 22:54:28.067797 4096 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 22:54:28.069309 4096 catalog_manager.cc:1347] Generated new cluster ID: 7db35e003e3f4cb9b1a8127919918112
I20260929 22:54:28.069360 4096 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 22:54:28.071991 4090 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:54:28.060281 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33947 (local address 127.14.147.254:42281)
0929 22:54:28.060458 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0929 22:54:28.060466 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 22:54:28.060492 (+ 26us) server_negotiation.cc:407] Connection header received
0929 22:54:28.061032 (+ 540us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 22:54:28.061042 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 22:54:28.061114 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 22:54:28.061267 (+ 153us) server_negotiation.cc:226] Negotiated authn=SASL
0929 22:54:28.061923 (+ 656us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:54:28.063355 (+ 1432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:54:28.064453 (+ 1098us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:54:28.065102 (+ 649us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:54:28.065215 (+ 113us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 22:54:28.068094 (+ 2879us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 22:54:28.068098 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 22:54:28.068099 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 22:54:28.068111 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0929 22:54:28.070124 (+ 2013us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 22:54:28.070539 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 22:54:28.070541 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 22:54:28.070542 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 22:54:28.070643 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 22:54:28.070836 (+ 193us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 22:54:28.070836 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 22:54:28.070837 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 22:54:28.071112 (+ 275us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 22:54:28.071261 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0929 22:54:28.071705 (+ 444us) server_negotiation.cc:299] Negotiation successful
0929 22:54:28.071797 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":127,"thread_start_us":55,"threads_started":1}
I20260929 22:54:28.072940 14927 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 22:54:28.075842 4096 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 22:54:28.076355 4096 catalog_manager.cc:1504] Loading token signing keys...
I20260929 22:54:28.079556 4096 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d1b8ae21083e4936aa9d1faf2df93d8b: Generated new TSK 0
I20260929 22:54:28.079756 4096 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 22:54:28.081211 4090 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:54:28.074356 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57547 (local address 127.14.147.254:42281)
0929 22:54:28.074388 (+ 32us) server_negotiation.cc:206] Beginning negotiation
0929 22:54:28.074390 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 22:54:28.074494 (+ 104us) server_negotiation.cc:407] Connection header received
0929 22:54:28.074556 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 22:54:28.074562 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 22:54:28.074572 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 22:54:28.074627 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0929 22:54:28.075252 (+ 625us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:54:28.075945 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:54:28.076934 (+ 989us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:54:28.077237 (+ 303us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:54:28.077301 (+ 64us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 22:54:28.078669 (+ 1368us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 22:54:28.078671 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 22:54:28.078671 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 22:54:28.078678 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0929 22:54:28.080188 (+ 1510us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 22:54:28.080589 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 22:54:28.080590 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 22:54:28.080590 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 22:54:28.080633 (+ 43us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 22:54:28.080817 (+ 184us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 22:54:28.080817 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 22:54:28.080817 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 22:54:28.080903 (+ 86us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 22:54:28.080968 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0929 22:54:28.081108 (+ 140us) server_negotiation.cc:299] Negotiation successful
0929 22:54:28.081153 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260929 22:54:28.083151 14927 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhWOCLk/build/release/bin/kudu with pid 4018
W20260929 22:54:28.086769 4003 connection.cc:535] client connection to 127.14.147.254:42281 recv error: Network error: failed to read from TLS socket (remote: 127.14.147.254:42281): Connection reset by peer (error 104)
2026-09-29T22:54:28Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/12 (1225 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/13
Loading random data
Initializing database '/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790722213368738-14927-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 22:54:28 dist-test-slave-98mk krb5kdc[4117](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 22:54:28 dist-test-slave-98mk krb5kdc[4117](info): set up 2 sockets
Sep 29 22:54:28 dist-test-slave-98mk krb5kdc[4117](info): commencing operation
krb5kdc: starting...
W20260929 22:54:29.032116 14927 mini_kdc.cc:121] Time spent starting KDC: real 0.932s user 0.002s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790722213368738-14927-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790722213368738-14927-0/krb5kdc/test-user.keytab.
Sep 29 22:54:29 dist-test-slave-98mk krb5kdc[4117](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790722469, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T22:54:29Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T22:54:29Z Disabled control of system clock
WARNING: no policy specified for kudu/127.14.147.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.147.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.147.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790722213368738-14927-0/krb5kdc/kudu_127.14.147.254.keytab.
Entry for principal kudu/127.14.147.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790722213368738-14927-0/krb5kdc/kudu_127.14.147.254.keytab.
I20260929 22:54:29.151497 14927 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhWOCLk/build/release/bin/kudu
/tmp/dist-test-taskhWOCLk/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790722213368738-14927-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790722213368738-14927-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790722213368738-14927-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790722213368738-14927-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.147.254:37047
--webserver_interface=127.14.147.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790722213368738-14927-0/krb5kdc/kudu_127.14.147.254.keytab
--principal=kudu/127.14.147.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.147.212:34029
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.147.254:37047
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790722213368738-14927-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790722213368738-14927-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790722213368738-14927-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 22:54:29.232707 4130 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 22:54:29.232939 4130 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 22:54:29.232961 4130 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 22:54:29.234134 4130 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 22:54:29.234182 4130 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 22:54:29.234195 4130 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 22:54:29.234206 4130 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 22:54:29.234217 4130 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 22:54:29.235787 4130 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.147.212:34029
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790722213368738-14927-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790722213368738-14927-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.14.147.254:37047
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790722213368738-14927-0/krb5kdc/kudu_127.14.147.254.keytab
--principal=kudu/127.14.147.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.147.254:37047
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790722213368738-14927-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.14.147.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4130
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790722213368738-14927-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 21:58:54 UTC on ccycloud.quasar-xguoti.root.comops.site
build id 4681390
I20260929 22:54:29.236029 4130 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 22:54:29.236214 4130 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 22:54:29.240944 4138 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 22:54:29.241065 4130 server_base.cc:1034] running on GCE node
W20260929 22:54:29.240959 4136 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 22:54:29.241173 4135 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 22:54:29.241423 4130 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 22:54:29.241741 4130 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 22:54:29.242929 4130 hybrid_clock.cc:648] HybridClock initialized: now 1790722469242906 us; error 38 us; skew 500 ppm
Sep 29 22:54:29 dist-test-slave-98mk krb5kdc[4117](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790722469, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.147.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 22:54:29.245472 4130 init.cc:377] Logged in from keytab as kudu/127.14.147.254@KRBTEST.COM (short username kudu)
I20260929 22:54:29.245954 4130 webserver.cc:460] Webserver started at http://127.14.147.254:43499/ using document root <none> and password file <none>
I20260929 22:54:29.246168 4130 fs_manager.cc:359] Metadata directory not provided
I20260929 22:54:29.246217 4130 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 22:54:29.246345 4130 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 22:54:29.247562 4130 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790722213368738-14927-0/minicluster-data/master-0/data/instance:
uuid: "28c9b2aacc1c4d3c9a36664c7630aef3"
format_stamp: "Formatted at 2026-09-29 22:54:29 on dist-test-slave-98mk"
server_key: "158058e74fac99c3dce11db447ec96ac"
server_key_iv: "bc99fc3d99a73ce8faf3277a8cc27f3d"
server_key_version: "encryptionkey@0"
I20260929 22:54:29.247905 4130 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790722213368738-14927-0/minicluster-data/master-0/wal/instance:
uuid: "28c9b2aacc1c4d3c9a36664c7630aef3"
format_stamp: "Formatted at 2026-09-29 22:54:29 on dist-test-slave-98mk"
server_key: "158058e74fac99c3dce11db447ec96ac"
server_key_iv: "bc99fc3d99a73ce8faf3277a8cc27f3d"
server_key_version: "encryptionkey@0"
I20260929 22:54:29.249375 4130 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260929 22:54:29.250527 4145 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 22:54:29.250746 4130 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260929 22:54:29.250850 4130 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790722213368738-14927-0/minicluster-data/master-0/data,/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790722213368738-14927-0/minicluster-data/master-0/wal
uuid: "28c9b2aacc1c4d3c9a36664c7630aef3"
format_stamp: "Formatted at 2026-09-29 22:54:29 on dist-test-slave-98mk"
server_key: "158058e74fac99c3dce11db447ec96ac"
server_key_iv: "bc99fc3d99a73ce8faf3277a8cc27f3d"
server_key_version: "encryptionkey@0"
I20260929 22:54:29.250926 4130 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790722213368738-14927-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790722213368738-14927-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790722213368738-14927-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 22:54:29.273711 4130 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 22:54:29.276185 4130 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 22:54:29.276346 4130 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 22:54:29.281281 4130 rpc_server.cc:307] RPC server started. Bound to: 127.14.147.254:37047
I20260929 22:54:29.281334 4197 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.147.254:37047 every 8 connection(s)
I20260929 22:54:29.281723 4130 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790722213368738-14927-0/minicluster-data/master-0/data/info.pb
I20260929 22:54:29.282724 4198 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 22:54:29.285885 4198 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 28c9b2aacc1c4d3c9a36664c7630aef3: Bootstrap starting.
I20260929 22:54:29.286620 4198 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 28c9b2aacc1c4d3c9a36664c7630aef3: Neither blocks nor log segments found. Creating new log.
I20260929 22:54:29.286652 14927 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhWOCLk/build/release/bin/kudu as pid 4130
I20260929 22:54:29.286772 14927 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790722213368738-14927-0/minicluster-data/master-0/wal/instance
I20260929 22:54:29.286944 4198 log.cc:824] T 00000000000000000000000000000000 P 28c9b2aacc1c4d3c9a36664c7630aef3: Log is configured to *not* fsync() on all Append() calls
I20260929 22:54:29.286981 14927 external_mini_cluster.cc:1428] Setting key 3faa72cd6586b3e9f6cb379e6dc6bc86
I20260929 22:54:29.287837 4198 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 28c9b2aacc1c4d3c9a36664c7630aef3: No bootstrap required, opened a new log
I20260929 22:54:29.291507 4198 raft_consensus.cc:348] T 00000000000000000000000000000000 P 28c9b2aacc1c4d3c9a36664c7630aef3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "28c9b2aacc1c4d3c9a36664c7630aef3" member_type: VOTER last_known_addr { host: "127.14.147.254" port: 37047 } }
I20260929 22:54:29.291718 4198 raft_consensus.cc:374] T 00000000000000000000000000000000 P 28c9b2aacc1c4d3c9a36664c7630aef3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 22:54:29.291752 4198 raft_consensus.cc:729] T 00000000000000000000000000000000 P 28c9b2aacc1c4d3c9a36664c7630aef3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 28c9b2aacc1c4d3c9a36664c7630aef3, State: Initialized, Role: FOLLOWER
I20260929 22:54:29.291887 4198 consensus_queue.cc:260] T 00000000000000000000000000000000 P 28c9b2aacc1c4d3c9a36664c7630aef3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "28c9b2aacc1c4d3c9a36664c7630aef3" member_type: VOTER last_known_addr { host: "127.14.147.254" port: 37047 } }
I20260929 22:54:29.291961 4198 raft_consensus.cc:388] T 00000000000000000000000000000000 P 28c9b2aacc1c4d3c9a36664c7630aef3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 22:54:29.292001 4198 raft_consensus.cc:482] T 00000000000000000000000000000000 P 28c9b2aacc1c4d3c9a36664c7630aef3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 22:54:29.292039 4198 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 28c9b2aacc1c4d3c9a36664c7630aef3 [term 0 FOLLOWER]: Advancing to term 1
I20260929 22:54:29.292726 4198 raft_consensus.cc:504] T 00000000000000000000000000000000 P 28c9b2aacc1c4d3c9a36664c7630aef3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "28c9b2aacc1c4d3c9a36664c7630aef3" member_type: VOTER last_known_addr { host: "127.14.147.254" port: 37047 } }
I20260929 22:54:29.292860 4198 leader_election.cc:302] T 00000000000000000000000000000000 P 28c9b2aacc1c4d3c9a36664c7630aef3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 28c9b2aacc1c4d3c9a36664c7630aef3; no voters:
I20260929 22:54:29.293054 4198 leader_election.cc:288] T 00000000000000000000000000000000 P 28c9b2aacc1c4d3c9a36664c7630aef3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 22:54:29.293080 4203 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 28c9b2aacc1c4d3c9a36664c7630aef3 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 22:54:29.293187 4203 raft_consensus.cc:686] T 00000000000000000000000000000000 P 28c9b2aacc1c4d3c9a36664c7630aef3 [term 1 LEADER]: Becoming Leader. State: Replica: 28c9b2aacc1c4d3c9a36664c7630aef3, State: Running, Role: LEADER
I20260929 22:54:29.293270 4203 consensus_queue.cc:237] T 00000000000000000000000000000000 P 28c9b2aacc1c4d3c9a36664c7630aef3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "28c9b2aacc1c4d3c9a36664c7630aef3" member_type: VOTER last_known_addr { host: "127.14.147.254" port: 37047 } }
I20260929 22:54:29.293329 4198 sys_catalog.cc:564] T 00000000000000000000000000000000 P 28c9b2aacc1c4d3c9a36664c7630aef3 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 22:54:29.293620 4204 sys_catalog.cc:455] T 00000000000000000000000000000000 P 28c9b2aacc1c4d3c9a36664c7630aef3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "28c9b2aacc1c4d3c9a36664c7630aef3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "28c9b2aacc1c4d3c9a36664c7630aef3" member_type: VOTER last_known_addr { host: "127.14.147.254" port: 37047 } } }
I20260929 22:54:29.293694 4204 sys_catalog.cc:458] T 00000000000000000000000000000000 P 28c9b2aacc1c4d3c9a36664c7630aef3 [sys.catalog]: This master's current role is: LEADER
I20260929 22:54:29.293838 4205 sys_catalog.cc:455] T 00000000000000000000000000000000 P 28c9b2aacc1c4d3c9a36664c7630aef3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 28c9b2aacc1c4d3c9a36664c7630aef3. Latest consensus state: current_term: 1 leader_uuid: "28c9b2aacc1c4d3c9a36664c7630aef3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "28c9b2aacc1c4d3c9a36664c7630aef3" member_type: VOTER last_known_addr { host: "127.14.147.254" port: 37047 } } }
I20260929 22:54:29.293897 4205 sys_catalog.cc:458] T 00000000000000000000000000000000 P 28c9b2aacc1c4d3c9a36664c7630aef3 [sys.catalog]: This master's current role is: LEADER
I20260929 22:54:29.294404 4202 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:54:29.287534 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49367 (local address 127.14.147.254:37047)
0929 22:54:29.287715 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0929 22:54:29.287724 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 22:54:29.287759 (+ 35us) server_negotiation.cc:407] Connection header received
0929 22:54:29.288344 (+ 585us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 22:54:29.288353 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 22:54:29.288419 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 22:54:29.288578 (+ 159us) server_negotiation.cc:226] Negotiated authn=SASL
0929 22:54:29.288992 (+ 414us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:54:29.290106 (+ 1114us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:54:29.291293 (+ 1187us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:54:29.291973 (+ 680us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:54:29.292093 (+ 120us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 22:54:29.293319 (+ 1226us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 22:54:29.293395 (+ 76us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 22:54:29.293396 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 22:54:29.293619 (+ 223us) server_negotiation.cc:922] calling sasl_server_start()
0929 22:54:29.293660 (+ 41us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 22:54:29.293675 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 22:54:29.293867 (+ 192us) server_negotiation.cc:1035] Waiting for connection context
0929 22:54:29.294181 (+ 314us) server_negotiation.cc:299] Negotiation successful
0929 22:54:29.294212 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":123,"thread_start_us":65,"threads_started":1}
W20260929 22:54:29.295225 4219 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 28c9b2aacc1c4d3c9a36664c7630aef3: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 22:54:29.295276 4219 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260929 22:54:29.295331 4207 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 22:54:29.295476 14927 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 22:54:29.295519 4207 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 22:54:29.297188 4207 catalog_manager.cc:1347] Generated new cluster ID: 4da8848d7738497db9aa068957e6ec71
I20260929 22:54:29.297243 4207 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 22:54:29.308568 4202 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:54:29.297072 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35769 (local address 127.14.147.254:37047)
0929 22:54:29.297122 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0929 22:54:29.297124 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 22:54:29.297245 (+ 121us) server_negotiation.cc:407] Connection header received
0929 22:54:29.297386 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 22:54:29.297387 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 22:54:29.297399 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 22:54:29.297472 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0929 22:54:29.297616 (+ 144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:54:29.298510 (+ 894us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:54:29.300460 (+ 1950us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:54:29.301705 (+ 1245us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:54:29.301822 (+ 117us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 22:54:29.308123 (+ 6301us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 22:54:29.308125 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 22:54:29.308125 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 22:54:29.308132 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0929 22:54:29.308151 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 22:54:29.308180 (+ 29us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 22:54:29.308217 (+ 37us) server_negotiation.cc:1035] Waiting for connection context
0929 22:54:29.308406 (+ 189us) server_negotiation.cc:299] Negotiation successful
0929 22:54:29.308445 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260929 22:54:29.314368 4207 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 22:54:29.315119 4207 catalog_manager.cc:1504] Loading token signing keys...
I20260929 22:54:29.323132 4207 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 28c9b2aacc1c4d3c9a36664c7630aef3: Generated new TSK 0
I20260929 22:54:29.323320 4207 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 22:54:29.329056 14927 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhWOCLk/build/release/bin/kudu with pid 4130
W20260929 22:54:29.334218 4115 connection.cc:535] client connection to 127.14.147.254:37047 recv error: Network error: failed to read from TLS socket (remote: 127.14.147.254:37047): Connection reset by peer (error 104)
2026-09-29T22:54:29Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/13 (1247 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/14
Loading random data
Initializing database '/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790722213368738-14927-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 22:54:29 dist-test-slave-98mk krb5kdc[4229](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 22:54:29 dist-test-slave-98mk krb5kdc[4229](info): set up 2 sockets
Sep 29 22:54:29 dist-test-slave-98mk krb5kdc[4229](info): commencing operation
krb5kdc: starting...
W20260929 22:54:30.276010 14927 mini_kdc.cc:121] Time spent starting KDC: real 0.928s user 0.001s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790722213368738-14927-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790722213368738-14927-0/krb5kdc/test-user.keytab.
Sep 29 22:54:30 dist-test-slave-98mk krb5kdc[4229](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790722470, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T22:54:30Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T22:54:30Z Disabled control of system clock
WARNING: no policy specified for kudu/127.14.147.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.147.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.147.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790722213368738-14927-0/krb5kdc/kudu_127.14.147.254.keytab.
Entry for principal kudu/127.14.147.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790722213368738-14927-0/krb5kdc/kudu_127.14.147.254.keytab.
I20260929 22:54:30.389317 14927 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhWOCLk/build/release/bin/kudu
/tmp/dist-test-taskhWOCLk/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790722213368738-14927-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790722213368738-14927-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790722213368738-14927-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790722213368738-14927-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.147.254:33519
--webserver_interface=127.14.147.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790722213368738-14927-0/krb5kdc/kudu_127.14.147.254.keytab
--principal=kudu/127.14.147.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.147.212:45035
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.147.254:33519
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790722213368738-14927-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790722213368738-14927-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790722213368738-14927-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 22:54:30.470724 4242 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 22:54:30.470950 4242 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 22:54:30.470983 4242 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 22:54:30.472229 4242 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 22:54:30.472278 4242 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 22:54:30.472297 4242 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 22:54:30.472314 4242 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 22:54:30.472329 4242 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 22:54:30.473863 4242 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.147.212:45035
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790722213368738-14927-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790722213368738-14927-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.14.147.254:33519
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790722213368738-14927-0/krb5kdc/kudu_127.14.147.254.keytab
--principal=kudu/127.14.147.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.147.254:33519
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790722213368738-14927-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.14.147.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4242
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790722213368738-14927-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 21:58:54 UTC on ccycloud.quasar-xguoti.root.comops.site
build id 4681390
I20260929 22:54:30.474120 4242 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 22:54:30.474327 4242 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 22:54:30.478863 4250 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 22:54:30.478916 4242 server_base.cc:1034] running on GCE node
W20260929 22:54:30.478863 4248 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 22:54:30.478868 4247 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 22:54:30.479305 4242 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 22:54:30.479615 4242 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 22:54:30.480772 4242 hybrid_clock.cc:648] HybridClock initialized: now 1790722470480755 us; error 31 us; skew 500 ppm
Sep 29 22:54:30 dist-test-slave-98mk krb5kdc[4229](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790722470, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.147.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 22:54:30.483155 4242 init.cc:377] Logged in from keytab as kudu/127.14.147.254@KRBTEST.COM (short username kudu)
I20260929 22:54:30.483629 4242 webserver.cc:460] Webserver started at http://127.14.147.254:36869/ using document root <none> and password file <none>
I20260929 22:54:30.483831 4242 fs_manager.cc:359] Metadata directory not provided
I20260929 22:54:30.483876 4242 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 22:54:30.484048 4242 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 22:54:30.485102 4242 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790722213368738-14927-0/minicluster-data/master-0/data/instance:
uuid: "9173a18d91a9461c8e84a7480347e83d"
format_stamp: "Formatted at 2026-09-29 22:54:30 on dist-test-slave-98mk"
server_key: "15f8a05077438add9240f28a742db105"
server_key_iv: "925b253591ad6ca683b7497103e64864"
server_key_version: "encryptionkey@0"
I20260929 22:54:30.485427 4242 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790722213368738-14927-0/minicluster-data/master-0/wal/instance:
uuid: "9173a18d91a9461c8e84a7480347e83d"
format_stamp: "Formatted at 2026-09-29 22:54:30 on dist-test-slave-98mk"
server_key: "15f8a05077438add9240f28a742db105"
server_key_iv: "925b253591ad6ca683b7497103e64864"
server_key_version: "encryptionkey@0"
I20260929 22:54:30.486626 4242 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260929 22:54:30.487259 4257 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 22:54:30.487396 4242 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260929 22:54:30.487497 4242 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790722213368738-14927-0/minicluster-data/master-0/data,/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790722213368738-14927-0/minicluster-data/master-0/wal
uuid: "9173a18d91a9461c8e84a7480347e83d"
format_stamp: "Formatted at 2026-09-29 22:54:30 on dist-test-slave-98mk"
server_key: "15f8a05077438add9240f28a742db105"
server_key_iv: "925b253591ad6ca683b7497103e64864"
server_key_version: "encryptionkey@0"
I20260929 22:54:30.487555 4242 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790722213368738-14927-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790722213368738-14927-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790722213368738-14927-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 22:54:30.490676 4242 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 22:54:30.492450 4242 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 22:54:30.492573 4242 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 22:54:30.496183 4242 rpc_server.cc:307] RPC server started. Bound to: 127.14.147.254:33519
I20260929 22:54:30.496212 4309 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.147.254:33519 every 8 connection(s)
I20260929 22:54:30.496510 4242 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790722213368738-14927-0/minicluster-data/master-0/data/info.pb
I20260929 22:54:30.497442 4310 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 22:54:30.499646 4310 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9173a18d91a9461c8e84a7480347e83d: Bootstrap starting.
I20260929 22:54:30.500183 4310 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9173a18d91a9461c8e84a7480347e83d: Neither blocks nor log segments found. Creating new log.
I20260929 22:54:30.500427 4310 log.cc:824] T 00000000000000000000000000000000 P 9173a18d91a9461c8e84a7480347e83d: Log is configured to *not* fsync() on all Append() calls
I20260929 22:54:30.501072 4310 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9173a18d91a9461c8e84a7480347e83d: No bootstrap required, opened a new log
I20260929 22:54:30.502619 4310 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9173a18d91a9461c8e84a7480347e83d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9173a18d91a9461c8e84a7480347e83d" member_type: VOTER last_known_addr { host: "127.14.147.254" port: 33519 } }
I20260929 22:54:30.502753 4310 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9173a18d91a9461c8e84a7480347e83d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 22:54:30.502785 4310 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9173a18d91a9461c8e84a7480347e83d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9173a18d91a9461c8e84a7480347e83d, State: Initialized, Role: FOLLOWER
I20260929 22:54:30.502924 4310 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9173a18d91a9461c8e84a7480347e83d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9173a18d91a9461c8e84a7480347e83d" member_type: VOTER last_known_addr { host: "127.14.147.254" port: 33519 } }
I20260929 22:54:30.502985 4310 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9173a18d91a9461c8e84a7480347e83d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 22:54:30.503005 4310 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9173a18d91a9461c8e84a7480347e83d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 22:54:30.503031 4310 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9173a18d91a9461c8e84a7480347e83d [term 0 FOLLOWER]: Advancing to term 1
I20260929 22:54:30.503896 4310 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9173a18d91a9461c8e84a7480347e83d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9173a18d91a9461c8e84a7480347e83d" member_type: VOTER last_known_addr { host: "127.14.147.254" port: 33519 } }
I20260929 22:54:30.503931 14927 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhWOCLk/build/release/bin/kudu as pid 4242
I20260929 22:54:30.504031 14927 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790722213368738-14927-0/minicluster-data/master-0/wal/instance
I20260929 22:54:30.504007 4310 leader_election.cc:302] T 00000000000000000000000000000000 P 9173a18d91a9461c8e84a7480347e83d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9173a18d91a9461c8e84a7480347e83d; no voters:
I20260929 22:54:30.504176 4310 leader_election.cc:288] T 00000000000000000000000000000000 P 9173a18d91a9461c8e84a7480347e83d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 22:54:30.504216 14927 external_mini_cluster.cc:1428] Setting key 3fd28a7a5d69a0f7b86ad8a05e079b2f
I20260929 22:54:30.504215 4313 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9173a18d91a9461c8e84a7480347e83d [term 1 FOLLOWER]: Leader election won for term 1
I20260929 22:54:30.504419 4310 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9173a18d91a9461c8e84a7480347e83d [sys.catalog]: configured and running, proceeding with master startup.
I20260929 22:54:30.504442 4313 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9173a18d91a9461c8e84a7480347e83d [term 1 LEADER]: Becoming Leader. State: Replica: 9173a18d91a9461c8e84a7480347e83d, State: Running, Role: LEADER
I20260929 22:54:30.504657 4313 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9173a18d91a9461c8e84a7480347e83d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9173a18d91a9461c8e84a7480347e83d" member_type: VOTER last_known_addr { host: "127.14.147.254" port: 33519 } }
I20260929 22:54:30.505091 4314 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9173a18d91a9461c8e84a7480347e83d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9173a18d91a9461c8e84a7480347e83d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9173a18d91a9461c8e84a7480347e83d" member_type: VOTER last_known_addr { host: "127.14.147.254" port: 33519 } } }
I20260929 22:54:30.505164 4314 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9173a18d91a9461c8e84a7480347e83d [sys.catalog]: This master's current role is: LEADER
I20260929 22:54:30.505349 4315 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9173a18d91a9461c8e84a7480347e83d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9173a18d91a9461c8e84a7480347e83d. Latest consensus state: current_term: 1 leader_uuid: "9173a18d91a9461c8e84a7480347e83d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9173a18d91a9461c8e84a7480347e83d" member_type: VOTER last_known_addr { host: "127.14.147.254" port: 33519 } } }
I20260929 22:54:30.505460 4315 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9173a18d91a9461c8e84a7480347e83d [sys.catalog]: This master's current role is: LEADER
I20260929 22:54:30.505626 4324 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 22:54:30.506155 4324 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 22:54:30.506372 4317 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:54:30.504635 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46539 (local address 127.14.147.254:33519)
0929 22:54:30.504792 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0929 22:54:30.504801 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 22:54:30.504834 (+ 33us) server_negotiation.cc:407] Connection header received
0929 22:54:30.504951 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 22:54:30.504958 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 22:54:30.504995 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 22:54:30.505421 (+ 426us) server_negotiation.cc:226] Negotiated authn=SASL
0929 22:54:30.505876 (+ 455us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 22:54:30.505879 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 22:54:30.505880 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 22:54:30.505883 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0929 22:54:30.505922 (+ 39us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 22:54:30.505937 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 22:54:30.506127 (+ 190us) server_negotiation.cc:1035] Waiting for connection context
0929 22:54:30.506159 (+ 32us) server_negotiation.cc:299] Negotiation successful
0929 22:54:30.506201 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":110,"thread_start_us":62,"threads_started":1}
I20260929 22:54:30.507673 14927 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 22:54:30.508787 4324 catalog_manager.cc:1347] Generated new cluster ID: d1f698a119324baea932fb39133e81ff
I20260929 22:54:30.508841 4324 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 22:54:30.520684 4317 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:54:30.519508 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46415 (local address 127.14.147.254:33519)
0929 22:54:30.519567 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0929 22:54:30.519569 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 22:54:30.519581 (+ 12us) server_negotiation.cc:407] Connection header received
0929 22:54:30.519613 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 22:54:30.519615 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 22:54:30.519624 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 22:54:30.519708 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0929 22:54:30.520438 (+ 730us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 22:54:30.520440 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 22:54:30.520440 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 22:54:30.520444 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0929 22:54:30.520461 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 22:54:30.520465 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 22:54:30.520517 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0929 22:54:30.520577 (+ 60us) server_negotiation.cc:299] Negotiation successful
0929 22:54:30.520591 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260929 22:54:30.523229 4324 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 22:54:30.523388 4324 catalog_manager.cc:1504] Loading token signing keys...
I20260929 22:54:30.529462 4324 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9173a18d91a9461c8e84a7480347e83d: Generated new TSK 0
I20260929 22:54:30.529670 4324 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 22:54:30.532038 14927 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhWOCLk/build/release/bin/kudu with pid 4242
2026-09-29T22:54:30Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/14 (1201 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/15
Loading random data
Initializing database '/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790722213368738-14927-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 22:54:30 dist-test-slave-98mk krb5kdc[4341](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 22:54:30 dist-test-slave-98mk krb5kdc[4341](info): set up 2 sockets
Sep 29 22:54:30 dist-test-slave-98mk krb5kdc[4341](info): commencing operation
krb5kdc: starting...
W20260929 22:54:31.484146 14927 mini_kdc.cc:121] Time spent starting KDC: real 0.935s user 0.002s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790722213368738-14927-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790722213368738-14927-0/krb5kdc/test-user.keytab.
Sep 29 22:54:31 dist-test-slave-98mk krb5kdc[4341](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790722471, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T22:54:31Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T22:54:31Z Disabled control of system clock
WARNING: no policy specified for kudu/127.14.147.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.147.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.147.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790722213368738-14927-0/krb5kdc/kudu_127.14.147.254.keytab.
Entry for principal kudu/127.14.147.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790722213368738-14927-0/krb5kdc/kudu_127.14.147.254.keytab.
I20260929 22:54:31.603788 14927 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhWOCLk/build/release/bin/kudu
/tmp/dist-test-taskhWOCLk/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790722213368738-14927-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790722213368738-14927-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790722213368738-14927-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790722213368738-14927-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.147.254:37013
--webserver_interface=127.14.147.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790722213368738-14927-0/krb5kdc/kudu_127.14.147.254.keytab
--principal=kudu/127.14.147.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.147.212:46321
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.147.254:37013
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790722213368738-14927-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790722213368738-14927-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790722213368738-14927-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 22:54:31.686271 4354 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 22:54:31.686512 4354 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 22:54:31.686556 4354 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 22:54:31.687892 4354 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 22:54:31.687945 4354 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 22:54:31.687965 4354 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 22:54:31.687981 4354 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 22:54:31.687996 4354 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 22:54:31.689831 4354 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.147.212:46321
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790722213368738-14927-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790722213368738-14927-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.14.147.254:37013
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790722213368738-14927-0/krb5kdc/kudu_127.14.147.254.keytab
--principal=kudu/127.14.147.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.147.254:37013
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790722213368738-14927-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.14.147.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4354
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790722213368738-14927-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 21:58:54 UTC on ccycloud.quasar-xguoti.root.comops.site
build id 4681390
I20260929 22:54:31.690124 4354 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 22:54:31.690346 4354 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 22:54:31.695209 4360 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 22:54:31.695217 4359 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 22:54:31.695217 4362 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 22:54:31.695578 4354 server_base.cc:1034] running on GCE node
I20260929 22:54:31.695806 4354 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 22:54:31.696080 4354 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 22:54:31.697224 4354 hybrid_clock.cc:648] HybridClock initialized: now 1790722471697212 us; error 31 us; skew 500 ppm
Sep 29 22:54:31 dist-test-slave-98mk krb5kdc[4341](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790722471, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.147.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 22:54:31.699643 4354 init.cc:377] Logged in from keytab as kudu/127.14.147.254@KRBTEST.COM (short username kudu)
I20260929 22:54:31.700078 4354 webserver.cc:460] Webserver started at http://127.14.147.254:44171/ using document root <none> and password file <none>
I20260929 22:54:31.700263 4354 fs_manager.cc:359] Metadata directory not provided
I20260929 22:54:31.700300 4354 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 22:54:31.700403 4354 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 22:54:31.701449 4354 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790722213368738-14927-0/minicluster-data/master-0/data/instance:
uuid: "30ef4969575d48f3a00dbbfbcb0f00b4"
format_stamp: "Formatted at 2026-09-29 22:54:31 on dist-test-slave-98mk"
server_key: "25f38ffbdbc0b2f75276ffa00fc341f6"
server_key_iv: "8599c1653b35ada3c13b700bbb0ab955"
server_key_version: "encryptionkey@0"
I20260929 22:54:31.701761 4354 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790722213368738-14927-0/minicluster-data/master-0/wal/instance:
uuid: "30ef4969575d48f3a00dbbfbcb0f00b4"
format_stamp: "Formatted at 2026-09-29 22:54:31 on dist-test-slave-98mk"
server_key: "25f38ffbdbc0b2f75276ffa00fc341f6"
server_key_iv: "8599c1653b35ada3c13b700bbb0ab955"
server_key_version: "encryptionkey@0"
I20260929 22:54:31.702981 4354 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260929 22:54:31.703671 4369 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 22:54:31.703840 4354 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260929 22:54:31.704043 4354 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790722213368738-14927-0/minicluster-data/master-0/data,/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790722213368738-14927-0/minicluster-data/master-0/wal
uuid: "30ef4969575d48f3a00dbbfbcb0f00b4"
format_stamp: "Formatted at 2026-09-29 22:54:31 on dist-test-slave-98mk"
server_key: "25f38ffbdbc0b2f75276ffa00fc341f6"
server_key_iv: "8599c1653b35ada3c13b700bbb0ab955"
server_key_version: "encryptionkey@0"
I20260929 22:54:31.704221 4354 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790722213368738-14927-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790722213368738-14927-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790722213368738-14927-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 22:54:31.717566 4354 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 22:54:31.719766 4354 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 22:54:31.719893 4354 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 22:54:31.723578 4354 rpc_server.cc:307] RPC server started. Bound to: 127.14.147.254:37013
I20260929 22:54:31.723639 4421 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.147.254:37013 every 8 connection(s)
I20260929 22:54:31.723948 4354 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790722213368738-14927-0/minicluster-data/master-0/data/info.pb
I20260929 22:54:31.724786 4422 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 22:54:31.727092 4422 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 30ef4969575d48f3a00dbbfbcb0f00b4: Bootstrap starting.
I20260929 22:54:31.727698 4422 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 30ef4969575d48f3a00dbbfbcb0f00b4: Neither blocks nor log segments found. Creating new log.
I20260929 22:54:31.727952 4422 log.cc:824] T 00000000000000000000000000000000 P 30ef4969575d48f3a00dbbfbcb0f00b4: Log is configured to *not* fsync() on all Append() calls
I20260929 22:54:31.728696 4422 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 30ef4969575d48f3a00dbbfbcb0f00b4: No bootstrap required, opened a new log
I20260929 22:54:31.729624 14927 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhWOCLk/build/release/bin/kudu as pid 4354
I20260929 22:54:31.729738 14927 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790722213368738-14927-0/minicluster-data/master-0/wal/instance
I20260929 22:54:31.729929 14927 external_mini_cluster.cc:1428] Setting key 0fd9a5d1f1ea98dd785cd58a25e96bdc
I20260929 22:54:31.730681 4422 raft_consensus.cc:348] T 00000000000000000000000000000000 P 30ef4969575d48f3a00dbbfbcb0f00b4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "30ef4969575d48f3a00dbbfbcb0f00b4" member_type: VOTER last_known_addr { host: "127.14.147.254" port: 37013 } }
I20260929 22:54:31.730829 4422 raft_consensus.cc:374] T 00000000000000000000000000000000 P 30ef4969575d48f3a00dbbfbcb0f00b4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 22:54:31.730850 4422 raft_consensus.cc:729] T 00000000000000000000000000000000 P 30ef4969575d48f3a00dbbfbcb0f00b4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 30ef4969575d48f3a00dbbfbcb0f00b4, State: Initialized, Role: FOLLOWER
I20260929 22:54:31.730963 4422 consensus_queue.cc:260] T 00000000000000000000000000000000 P 30ef4969575d48f3a00dbbfbcb0f00b4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "30ef4969575d48f3a00dbbfbcb0f00b4" member_type: VOTER last_known_addr { host: "127.14.147.254" port: 37013 } }
I20260929 22:54:31.731035 4422 raft_consensus.cc:388] T 00000000000000000000000000000000 P 30ef4969575d48f3a00dbbfbcb0f00b4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 22:54:31.731066 4422 raft_consensus.cc:482] T 00000000000000000000000000000000 P 30ef4969575d48f3a00dbbfbcb0f00b4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 22:54:31.731145 4422 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 30ef4969575d48f3a00dbbfbcb0f00b4 [term 0 FOLLOWER]: Advancing to term 1
I20260929 22:54:31.732080 4422 raft_consensus.cc:504] T 00000000000000000000000000000000 P 30ef4969575d48f3a00dbbfbcb0f00b4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "30ef4969575d48f3a00dbbfbcb0f00b4" member_type: VOTER last_known_addr { host: "127.14.147.254" port: 37013 } }
I20260929 22:54:31.732185 4422 leader_election.cc:302] T 00000000000000000000000000000000 P 30ef4969575d48f3a00dbbfbcb0f00b4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 30ef4969575d48f3a00dbbfbcb0f00b4; no voters:
I20260929 22:54:31.732345 4422 leader_election.cc:288] T 00000000000000000000000000000000 P 30ef4969575d48f3a00dbbfbcb0f00b4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 22:54:31.732379 4427 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 30ef4969575d48f3a00dbbfbcb0f00b4 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 22:54:31.732484 4427 raft_consensus.cc:686] T 00000000000000000000000000000000 P 30ef4969575d48f3a00dbbfbcb0f00b4 [term 1 LEADER]: Becoming Leader. State: Replica: 30ef4969575d48f3a00dbbfbcb0f00b4, State: Running, Role: LEADER
I20260929 22:54:31.732635 4422 sys_catalog.cc:564] T 00000000000000000000000000000000 P 30ef4969575d48f3a00dbbfbcb0f00b4 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 22:54:31.732622 4427 consensus_queue.cc:237] T 00000000000000000000000000000000 P 30ef4969575d48f3a00dbbfbcb0f00b4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "30ef4969575d48f3a00dbbfbcb0f00b4" member_type: VOTER last_known_addr { host: "127.14.147.254" port: 37013 } }
I20260929 22:54:31.732957 4427 sys_catalog.cc:455] T 00000000000000000000000000000000 P 30ef4969575d48f3a00dbbfbcb0f00b4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 30ef4969575d48f3a00dbbfbcb0f00b4. Latest consensus state: current_term: 1 leader_uuid: "30ef4969575d48f3a00dbbfbcb0f00b4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "30ef4969575d48f3a00dbbfbcb0f00b4" member_type: VOTER last_known_addr { host: "127.14.147.254" port: 37013 } } }
I20260929 22:54:31.733036 4427 sys_catalog.cc:458] T 00000000000000000000000000000000 P 30ef4969575d48f3a00dbbfbcb0f00b4 [sys.catalog]: This master's current role is: LEADER
I20260929 22:54:31.733186 4428 sys_catalog.cc:455] T 00000000000000000000000000000000 P 30ef4969575d48f3a00dbbfbcb0f00b4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "30ef4969575d48f3a00dbbfbcb0f00b4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "30ef4969575d48f3a00dbbfbcb0f00b4" member_type: VOTER last_known_addr { host: "127.14.147.254" port: 37013 } } }
I20260929 22:54:31.733235 4433 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 22:54:31.733331 4428 sys_catalog.cc:458] T 00000000000000000000000000000000 P 30ef4969575d48f3a00dbbfbcb0f00b4 [sys.catalog]: This master's current role is: LEADER
I20260929 22:54:31.734611 4433 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 22:54:31.736290 4433 catalog_manager.cc:1347] Generated new cluster ID: 8a8981dd88c64ba288aa67a9323db34b
I20260929 22:54:31.736346 4433 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 29 22:54:31 dist-test-slave-98mk krb5kdc[4341](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790722471, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.147.254@KRBTEST.COM
I20260929 22:54:31.740365 4426 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:54:31.730478 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35657 (local address 127.14.147.254:37013)
0929 22:54:31.730668 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0929 22:54:31.730676 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 22:54:31.730702 (+ 26us) server_negotiation.cc:407] Connection header received
0929 22:54:31.730796 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 22:54:31.730802 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 22:54:31.730854 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 22:54:31.731030 (+ 176us) server_negotiation.cc:226] Negotiated authn=SASL
0929 22:54:31.731717 (+ 687us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:54:31.732862 (+ 1145us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:54:31.733846 (+ 984us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:54:31.734661 (+ 815us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:54:31.735131 (+ 470us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 22:54:31.737303 (+ 2172us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 22:54:31.737307 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 22:54:31.737308 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 22:54:31.737320 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0929 22:54:31.738877 (+ 1557us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 22:54:31.739215 (+ 338us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 22:54:31.739218 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 22:54:31.739218 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 22:54:31.739328 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 22:54:31.739541 (+ 213us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 22:54:31.739541 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 22:54:31.739542 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 22:54:31.739790 (+ 248us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 22:54:31.739948 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0929 22:54:31.740027 (+ 79us) server_negotiation.cc:299] Negotiation successful
0929 22:54:31.740167 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":142,"thread_start_us":114,"threads_started":1}
I20260929 22:54:31.741292 14927 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 22:54:31.746536 4433 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 22:54:31.747255 4433 catalog_manager.cc:1504] Loading token signing keys...
I20260929 22:54:31.753827 4426 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:54:31.746311 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60125 (local address 127.14.147.254:37013)
0929 22:54:31.746352 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0929 22:54:31.746354 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 22:54:31.746556 (+ 202us) server_negotiation.cc:407] Connection header received
0929 22:54:31.746587 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 22:54:31.746588 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 22:54:31.746601 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 22:54:31.746671 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0929 22:54:31.747409 (+ 738us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:54:31.748168 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:54:31.749234 (+ 1066us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:54:31.749571 (+ 337us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:54:31.749625 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 22:54:31.751060 (+ 1435us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 22:54:31.751061 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 22:54:31.751062 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 22:54:31.751076 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0929 22:54:31.752659 (+ 1583us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 22:54:31.753062 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 22:54:31.753063 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 22:54:31.753063 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 22:54:31.753125 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 22:54:31.753313 (+ 188us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 22:54:31.753313 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 22:54:31.753314 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 22:54:31.753429 (+ 115us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 22:54:31.753508 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0929 22:54:31.753636 (+ 128us) server_negotiation.cc:299] Negotiation successful
0929 22:54:31.753729 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260929 22:54:31.757094 4433 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 30ef4969575d48f3a00dbbfbcb0f00b4: Generated new TSK 0
I20260929 22:54:31.757315 4433 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 22:54:31.761652 14927 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhWOCLk/build/release/bin/kudu with pid 4354
2026-09-29T22:54:31Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/15 (1229 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/16
Loading random data
Initializing database '/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790722213368738-14927-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 22:54:31 dist-test-slave-98mk krb5kdc[4453](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 22:54:31 dist-test-slave-98mk krb5kdc[4453](info): set up 2 sockets
Sep 29 22:54:31 dist-test-slave-98mk krb5kdc[4453](info): commencing operation
krb5kdc: starting...
W20260929 22:54:32.720003 14927 mini_kdc.cc:121] Time spent starting KDC: real 0.941s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790722213368738-14927-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790722213368738-14927-0/krb5kdc/test-user.keytab.
Sep 29 22:54:32 dist-test-slave-98mk krb5kdc[4453](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790722472, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T22:54:32Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T22:54:32Z Disabled control of system clock
WARNING: no policy specified for kudu/127.14.147.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.147.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.147.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790722213368738-14927-0/krb5kdc/kudu_127.14.147.254.keytab.
Entry for principal kudu/127.14.147.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790722213368738-14927-0/krb5kdc/kudu_127.14.147.254.keytab.
I20260929 22:54:32.834337 14927 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhWOCLk/build/release/bin/kudu
/tmp/dist-test-taskhWOCLk/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790722213368738-14927-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790722213368738-14927-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790722213368738-14927-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790722213368738-14927-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.147.254:43647
--webserver_interface=127.14.147.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790722213368738-14927-0/krb5kdc/kudu_127.14.147.254.keytab
--principal=kudu/127.14.147.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.147.212:44031
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.147.254:43647
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790722213368738-14927-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790722213368738-14927-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790722213368738-14927-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 22:54:32.914469 4466 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 22:54:32.914680 4466 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 22:54:32.914713 4466 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 22:54:32.915910 4466 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 22:54:32.915957 4466 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 22:54:32.915977 4466 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 22:54:32.915992 4466 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 22:54:32.916007 4466 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 22:54:32.917490 4466 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.147.212:44031
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790722213368738-14927-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790722213368738-14927-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.14.147.254:43647
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790722213368738-14927-0/krb5kdc/kudu_127.14.147.254.keytab
--principal=kudu/127.14.147.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.147.254:43647
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790722213368738-14927-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.14.147.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4466
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790722213368738-14927-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 21:58:54 UTC on ccycloud.quasar-xguoti.root.comops.site
build id 4681390
I20260929 22:54:32.917730 4466 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 22:54:32.917950 4466 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 22:54:32.922454 4471 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 22:54:32.922444 4472 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 22:54:32.922446 4474 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 22:54:32.922626 4466 server_base.cc:1034] running on GCE node
I20260929 22:54:32.922838 4466 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 22:54:32.923115 4466 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 22:54:32.924255 4466 hybrid_clock.cc:648] HybridClock initialized: now 1790722472924238 us; error 30 us; skew 500 ppm
Sep 29 22:54:32 dist-test-slave-98mk krb5kdc[4453](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790722472, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.147.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 22:54:32.926596 4466 init.cc:377] Logged in from keytab as kudu/127.14.147.254@KRBTEST.COM (short username kudu)
I20260929 22:54:32.927006 4466 webserver.cc:460] Webserver started at http://127.14.147.254:36081/ using document root <none> and password file <none>
I20260929 22:54:32.927228 4466 fs_manager.cc:359] Metadata directory not provided
I20260929 22:54:32.927276 4466 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 22:54:32.927397 4466 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 22:54:32.928464 4466 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790722213368738-14927-0/minicluster-data/master-0/data/instance:
uuid: "1bdc3e34f11f463292d8efe5c3694125"
format_stamp: "Formatted at 2026-09-29 22:54:32 on dist-test-slave-98mk"
server_key: "78ded3b92f15fa0c59af07546334708d"
server_key_iv: "1cc9735622cce94ff0cc935002041ec0"
server_key_version: "encryptionkey@0"
I20260929 22:54:32.928792 4466 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790722213368738-14927-0/minicluster-data/master-0/wal/instance:
uuid: "1bdc3e34f11f463292d8efe5c3694125"
format_stamp: "Formatted at 2026-09-29 22:54:32 on dist-test-slave-98mk"
server_key: "78ded3b92f15fa0c59af07546334708d"
server_key_iv: "1cc9735622cce94ff0cc935002041ec0"
server_key_version: "encryptionkey@0"
I20260929 22:54:32.930080 4466 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260929 22:54:32.930743 4481 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 22:54:32.930912 4466 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260929 22:54:32.930992 4466 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790722213368738-14927-0/minicluster-data/master-0/data,/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790722213368738-14927-0/minicluster-data/master-0/wal
uuid: "1bdc3e34f11f463292d8efe5c3694125"
format_stamp: "Formatted at 2026-09-29 22:54:32 on dist-test-slave-98mk"
server_key: "78ded3b92f15fa0c59af07546334708d"
server_key_iv: "1cc9735622cce94ff0cc935002041ec0"
server_key_version: "encryptionkey@0"
I20260929 22:54:32.931062 4466 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790722213368738-14927-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790722213368738-14927-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790722213368738-14927-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 22:54:32.947283 4466 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 22:54:32.949193 4466 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 22:54:32.949333 4466 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 22:54:32.953047 4466 rpc_server.cc:307] RPC server started. Bound to: 127.14.147.254:43647
I20260929 22:54:32.953150 4533 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.147.254:43647 every 8 connection(s)
I20260929 22:54:32.953431 4466 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790722213368738-14927-0/minicluster-data/master-0/data/info.pb
I20260929 22:54:32.954629 4534 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 22:54:32.957271 4534 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1bdc3e34f11f463292d8efe5c3694125: Bootstrap starting.
I20260929 22:54:32.957835 4534 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1bdc3e34f11f463292d8efe5c3694125: Neither blocks nor log segments found. Creating new log.
I20260929 22:54:32.958063 4534 log.cc:824] T 00000000000000000000000000000000 P 1bdc3e34f11f463292d8efe5c3694125: Log is configured to *not* fsync() on all Append() calls
I20260929 22:54:32.958731 4534 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1bdc3e34f11f463292d8efe5c3694125: No bootstrap required, opened a new log
I20260929 22:54:32.959398 14927 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhWOCLk/build/release/bin/kudu as pid 4466
I20260929 22:54:32.959558 14927 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790722213368738-14927-0/minicluster-data/master-0/wal/instance
I20260929 22:54:32.959714 14927 external_mini_cluster.cc:1428] Setting key 52f4f993053fd02673852d7e491e5aa7
I20260929 22:54:32.960760 4534 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1bdc3e34f11f463292d8efe5c3694125 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1bdc3e34f11f463292d8efe5c3694125" member_type: VOTER last_known_addr { host: "127.14.147.254" port: 43647 } }
I20260929 22:54:32.960935 4534 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1bdc3e34f11f463292d8efe5c3694125 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 22:54:32.960970 4534 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1bdc3e34f11f463292d8efe5c3694125 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1bdc3e34f11f463292d8efe5c3694125, State: Initialized, Role: FOLLOWER
I20260929 22:54:32.961124 4534 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1bdc3e34f11f463292d8efe5c3694125 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1bdc3e34f11f463292d8efe5c3694125" member_type: VOTER last_known_addr { host: "127.14.147.254" port: 43647 } }
I20260929 22:54:32.961203 4534 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1bdc3e34f11f463292d8efe5c3694125 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 22:54:32.961233 4534 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1bdc3e34f11f463292d8efe5c3694125 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 22:54:32.961282 4534 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1bdc3e34f11f463292d8efe5c3694125 [term 0 FOLLOWER]: Advancing to term 1
I20260929 22:54:32.961844 4534 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1bdc3e34f11f463292d8efe5c3694125 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1bdc3e34f11f463292d8efe5c3694125" member_type: VOTER last_known_addr { host: "127.14.147.254" port: 43647 } }
I20260929 22:54:32.961934 4534 leader_election.cc:302] T 00000000000000000000000000000000 P 1bdc3e34f11f463292d8efe5c3694125 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1bdc3e34f11f463292d8efe5c3694125; no voters:
I20260929 22:54:32.962086 4534 leader_election.cc:288] T 00000000000000000000000000000000 P 1bdc3e34f11f463292d8efe5c3694125 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 22:54:32.962114 4539 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1bdc3e34f11f463292d8efe5c3694125 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 22:54:32.962208 4539 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1bdc3e34f11f463292d8efe5c3694125 [term 1 LEADER]: Becoming Leader. State: Replica: 1bdc3e34f11f463292d8efe5c3694125, State: Running, Role: LEADER
I20260929 22:54:32.962340 4534 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1bdc3e34f11f463292d8efe5c3694125 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 22:54:32.962338 4539 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1bdc3e34f11f463292d8efe5c3694125 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1bdc3e34f11f463292d8efe5c3694125" member_type: VOTER last_known_addr { host: "127.14.147.254" port: 43647 } }
I20260929 22:54:32.962683 4541 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1bdc3e34f11f463292d8efe5c3694125 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1bdc3e34f11f463292d8efe5c3694125. Latest consensus state: current_term: 1 leader_uuid: "1bdc3e34f11f463292d8efe5c3694125" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1bdc3e34f11f463292d8efe5c3694125" member_type: VOTER last_known_addr { host: "127.14.147.254" port: 43647 } } }
I20260929 22:54:32.962790 4541 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1bdc3e34f11f463292d8efe5c3694125 [sys.catalog]: This master's current role is: LEADER
I20260929 22:54:32.963701 4540 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1bdc3e34f11f463292d8efe5c3694125 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1bdc3e34f11f463292d8efe5c3694125" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1bdc3e34f11f463292d8efe5c3694125" member_type: VOTER last_known_addr { host: "127.14.147.254" port: 43647 } } }
I20260929 22:54:32.963783 4540 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1bdc3e34f11f463292d8efe5c3694125 [sys.catalog]: This master's current role is: LEADER
I20260929 22:54:32.964015 4543 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 22:54:32.964561 4543 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 22:54:32.965885 4538 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:54:32.960220 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50933 (local address 127.14.147.254:43647)
0929 22:54:32.960453 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0929 22:54:32.960464 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0929 22:54:32.960502 (+ 38us) server_negotiation.cc:407] Connection header received
0929 22:54:32.960733 (+ 231us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 22:54:32.960742 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 22:54:32.960804 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 22:54:32.960949 (+ 145us) server_negotiation.cc:226] Negotiated authn=SASL
0929 22:54:32.961151 (+ 202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:54:32.962502 (+ 1351us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:54:32.963538 (+ 1036us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:54:32.964150 (+ 612us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:54:32.964253 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 22:54:32.965071 (+ 818us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 22:54:32.965078 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 22:54:32.965079 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 22:54:32.965376 (+ 297us) server_negotiation.cc:922] calling sasl_server_start()
0929 22:54:32.965424 (+ 48us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 22:54:32.965438 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 22:54:32.965525 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0929 22:54:32.965630 (+ 105us) server_negotiation.cc:299] Negotiation successful
0929 22:54:32.965697 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":156,"thread_start_us":101,"threads_started":1}
I20260929 22:54:32.966536 4543 catalog_manager.cc:1347] Generated new cluster ID: 7a9dd0695d9c4453b069121440e60d60
I20260929 22:54:32.966616 4543 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 22:54:32.966922 14927 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 22:54:32.974318 4538 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:54:32.971688 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42593 (local address 127.14.147.254:43647)
0929 22:54:32.971728 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0929 22:54:32.971730 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 22:54:32.971896 (+ 166us) server_negotiation.cc:407] Connection header received
0929 22:54:32.971939 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 22:54:32.971940 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 22:54:32.971951 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 22:54:32.971996 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0929 22:54:32.972119 (+ 123us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:54:32.972581 (+ 462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:54:32.973249 (+ 668us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:54:32.973433 (+ 184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:54:32.973487 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 22:54:32.974084 (+ 597us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 22:54:32.974085 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 22:54:32.974086 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 22:54:32.974090 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0929 22:54:32.974105 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 22:54:32.974109 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 22:54:32.974153 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0929 22:54:32.974215 (+ 62us) server_negotiation.cc:299] Negotiation successful
0929 22:54:32.974238 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260929 22:54:32.984158 4543 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 22:54:32.984695 4543 catalog_manager.cc:1504] Loading token signing keys...
I20260929 22:54:32.987826 4543 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1bdc3e34f11f463292d8efe5c3694125: Generated new TSK 0
I20260929 22:54:32.988008 4543 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 22:54:32.991571 14927 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhWOCLk/build/release/bin/kudu with pid 4466
2026-09-29T22:54:32Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/16 (1230 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/17
Loading random data
Initializing database '/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790722213368738-14927-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 22:54:33 dist-test-slave-98mk krb5kdc[4565](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 22:54:33 dist-test-slave-98mk krb5kdc[4565](info): set up 2 sockets
Sep 29 22:54:33 dist-test-slave-98mk krb5kdc[4565](info): commencing operation
krb5kdc: starting...
W20260929 22:54:33.940070 14927 mini_kdc.cc:121] Time spent starting KDC: real 0.931s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790722213368738-14927-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790722213368738-14927-0/krb5kdc/test-user.keytab.
Sep 29 22:54:34 dist-test-slave-98mk krb5kdc[4565](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790722474, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T22:54:34Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T22:54:34Z Disabled control of system clock
WARNING: no policy specified for kudu/127.14.147.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.147.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.147.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790722213368738-14927-0/krb5kdc/kudu_127.14.147.254.keytab.
Entry for principal kudu/127.14.147.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790722213368738-14927-0/krb5kdc/kudu_127.14.147.254.keytab.
I20260929 22:54:34.062055 14927 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhWOCLk/build/release/bin/kudu
/tmp/dist-test-taskhWOCLk/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790722213368738-14927-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790722213368738-14927-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790722213368738-14927-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790722213368738-14927-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.147.254:43223
--webserver_interface=127.14.147.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790722213368738-14927-0/krb5kdc/kudu_127.14.147.254.keytab
--principal=kudu/127.14.147.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.147.212:37069
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.147.254:43223
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790722213368738-14927-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790722213368738-14927-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790722213368738-14927-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 22:54:34.142324 4578 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 22:54:34.142563 4578 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 22:54:34.142599 4578 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 22:54:34.143880 4578 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 22:54:34.143927 4578 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 22:54:34.143947 4578 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 22:54:34.143963 4578 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 22:54:34.143980 4578 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 22:54:34.145581 4578 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.147.212:37069
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790722213368738-14927-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790722213368738-14927-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.14.147.254:43223
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790722213368738-14927-0/krb5kdc/kudu_127.14.147.254.keytab
--principal=kudu/127.14.147.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.147.254:43223
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790722213368738-14927-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.14.147.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4578
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790722213368738-14927-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 21:58:54 UTC on ccycloud.quasar-xguoti.root.comops.site
build id 4681390
I20260929 22:54:34.145846 4578 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 22:54:34.146059 4578 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 22:54:34.150746 4583 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 22:54:34.150753 4584 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 22:54:34.150774 4586 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 22:54:34.150949 4578 server_base.cc:1034] running on GCE node
I20260929 22:54:34.151190 4578 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 22:54:34.151522 4578 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 22:54:34.152680 4578 hybrid_clock.cc:648] HybridClock initialized: now 1790722474152660 us; error 36 us; skew 500 ppm
Sep 29 22:54:34 dist-test-slave-98mk krb5kdc[4565](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790722474, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.147.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 22:54:34.155375 4578 init.cc:377] Logged in from keytab as kudu/127.14.147.254@KRBTEST.COM (short username kudu)
I20260929 22:54:34.155890 4578 webserver.cc:460] Webserver started at http://127.14.147.254:44425/ using document root <none> and password file <none>
I20260929 22:54:34.156101 4578 fs_manager.cc:359] Metadata directory not provided
I20260929 22:54:34.156147 4578 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 22:54:34.156281 4578 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 22:54:34.157393 4578 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790722213368738-14927-0/minicluster-data/master-0/data/instance:
uuid: "9379ae80da294cf79fcb466c099f2a3c"
format_stamp: "Formatted at 2026-09-29 22:54:34 on dist-test-slave-98mk"
server_key: "07a7fb359df1a04983aec0f9f9fd0eab"
server_key_iv: "a939e4b9f98def0ebb6e06768177b2f5"
server_key_version: "encryptionkey@0"
I20260929 22:54:34.157730 4578 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790722213368738-14927-0/minicluster-data/master-0/wal/instance:
uuid: "9379ae80da294cf79fcb466c099f2a3c"
format_stamp: "Formatted at 2026-09-29 22:54:34 on dist-test-slave-98mk"
server_key: "07a7fb359df1a04983aec0f9f9fd0eab"
server_key_iv: "a939e4b9f98def0ebb6e06768177b2f5"
server_key_version: "encryptionkey@0"
I20260929 22:54:34.159020 4578 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260929 22:54:34.159765 4593 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 22:54:34.159910 4578 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260929 22:54:34.159986 4578 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790722213368738-14927-0/minicluster-data/master-0/data,/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790722213368738-14927-0/minicluster-data/master-0/wal
uuid: "9379ae80da294cf79fcb466c099f2a3c"
format_stamp: "Formatted at 2026-09-29 22:54:34 on dist-test-slave-98mk"
server_key: "07a7fb359df1a04983aec0f9f9fd0eab"
server_key_iv: "a939e4b9f98def0ebb6e06768177b2f5"
server_key_version: "encryptionkey@0"
I20260929 22:54:34.160055 4578 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790722213368738-14927-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790722213368738-14927-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790722213368738-14927-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 22:54:34.163301 4578 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 22:54:34.165418 4578 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 22:54:34.165547 4578 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 22:54:34.169325 4578 rpc_server.cc:307] RPC server started. Bound to: 127.14.147.254:43223
I20260929 22:54:34.169366 4645 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.147.254:43223 every 8 connection(s)
I20260929 22:54:34.169667 4578 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790722213368738-14927-0/minicluster-data/master-0/data/info.pb
I20260929 22:54:34.170614 4646 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 22:54:34.172930 4646 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9379ae80da294cf79fcb466c099f2a3c: Bootstrap starting.
I20260929 22:54:34.173492 4646 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9379ae80da294cf79fcb466c099f2a3c: Neither blocks nor log segments found. Creating new log.
I20260929 22:54:34.173754 4646 log.cc:824] T 00000000000000000000000000000000 P 9379ae80da294cf79fcb466c099f2a3c: Log is configured to *not* fsync() on all Append() calls
I20260929 22:54:34.174691 4646 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9379ae80da294cf79fcb466c099f2a3c: No bootstrap required, opened a new log
I20260929 22:54:34.176800 4646 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9379ae80da294cf79fcb466c099f2a3c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9379ae80da294cf79fcb466c099f2a3c" member_type: VOTER last_known_addr { host: "127.14.147.254" port: 43223 } }
I20260929 22:54:34.176992 4646 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9379ae80da294cf79fcb466c099f2a3c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 22:54:34.177016 4646 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9379ae80da294cf79fcb466c099f2a3c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9379ae80da294cf79fcb466c099f2a3c, State: Initialized, Role: FOLLOWER
I20260929 22:54:34.177033 14927 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhWOCLk/build/release/bin/kudu as pid 4578
I20260929 22:54:34.177143 14927 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790722213368738-14927-0/minicluster-data/master-0/wal/instance
I20260929 22:54:34.177167 4646 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9379ae80da294cf79fcb466c099f2a3c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9379ae80da294cf79fcb466c099f2a3c" member_type: VOTER last_known_addr { host: "127.14.147.254" port: 43223 } }
I20260929 22:54:34.177253 4646 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9379ae80da294cf79fcb466c099f2a3c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 22:54:34.177294 4646 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9379ae80da294cf79fcb466c099f2a3c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 22:54:34.177342 4646 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9379ae80da294cf79fcb466c099f2a3c [term 0 FOLLOWER]: Advancing to term 1
I20260929 22:54:34.177348 14927 external_mini_cluster.cc:1428] Setting key 2d8dd11fb7db8a63a984ead3d3d72481
I20260929 22:54:34.178542 4646 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9379ae80da294cf79fcb466c099f2a3c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9379ae80da294cf79fcb466c099f2a3c" member_type: VOTER last_known_addr { host: "127.14.147.254" port: 43223 } }
I20260929 22:54:34.178661 4646 leader_election.cc:302] T 00000000000000000000000000000000 P 9379ae80da294cf79fcb466c099f2a3c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9379ae80da294cf79fcb466c099f2a3c; no voters:
I20260929 22:54:34.178865 4646 leader_election.cc:288] T 00000000000000000000000000000000 P 9379ae80da294cf79fcb466c099f2a3c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 22:54:34.178920 4651 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9379ae80da294cf79fcb466c099f2a3c [term 1 FOLLOWER]: Leader election won for term 1
I20260929 22:54:34.179082 4651 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9379ae80da294cf79fcb466c099f2a3c [term 1 LEADER]: Becoming Leader. State: Replica: 9379ae80da294cf79fcb466c099f2a3c, State: Running, Role: LEADER
I20260929 22:54:34.179193 4646 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9379ae80da294cf79fcb466c099f2a3c [sys.catalog]: configured and running, proceeding with master startup.
I20260929 22:54:34.179219 4651 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9379ae80da294cf79fcb466c099f2a3c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9379ae80da294cf79fcb466c099f2a3c" member_type: VOTER last_known_addr { host: "127.14.147.254" port: 43223 } }
I20260929 22:54:34.179698 4650 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:54:34.177846 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40961 (local address 127.14.147.254:43223)
0929 22:54:34.178133 (+ 287us) server_negotiation.cc:206] Beginning negotiation
0929 22:54:34.178144 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0929 22:54:34.178184 (+ 40us) server_negotiation.cc:407] Connection header received
0929 22:54:34.178308 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 22:54:34.178316 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 22:54:34.178356 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 22:54:34.178509 (+ 153us) server_negotiation.cc:226] Negotiated authn=SASL
0929 22:54:34.179126 (+ 617us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 22:54:34.179130 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 22:54:34.179130 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 22:54:34.179135 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0929 22:54:34.179181 (+ 46us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 22:54:34.179197 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 22:54:34.179252 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0929 22:54:34.179446 (+ 194us) server_negotiation.cc:299] Negotiation successful
0929 22:54:34.179492 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":222,"thread_start_us":165,"threads_started":1}
I20260929 22:54:34.179675 4652 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9379ae80da294cf79fcb466c099f2a3c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9379ae80da294cf79fcb466c099f2a3c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9379ae80da294cf79fcb466c099f2a3c" member_type: VOTER last_known_addr { host: "127.14.147.254" port: 43223 } } }
I20260929 22:54:34.179945 4652 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9379ae80da294cf79fcb466c099f2a3c [sys.catalog]: This master's current role is: LEADER
I20260929 22:54:34.179754 4653 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9379ae80da294cf79fcb466c099f2a3c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9379ae80da294cf79fcb466c099f2a3c. Latest consensus state: current_term: 1 leader_uuid: "9379ae80da294cf79fcb466c099f2a3c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9379ae80da294cf79fcb466c099f2a3c" member_type: VOTER last_known_addr { host: "127.14.147.254" port: 43223 } } }
I20260929 22:54:34.180025 4653 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9379ae80da294cf79fcb466c099f2a3c [sys.catalog]: This master's current role is: LEADER
I20260929 22:54:34.180469 4660 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 22:54:34.181031 4660 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 22:54:34.181174 14927 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 22:54:34.183904 4660 catalog_manager.cc:1347] Generated new cluster ID: ad27e90f401f45049294acf3e851192a
I20260929 22:54:34.183979 4660 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 22:54:34.184957 4650 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:54:34.182747 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51879 (local address 127.14.147.254:43223)
0929 22:54:34.182788 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0929 22:54:34.182790 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 22:54:34.183989 (+ 1199us) server_negotiation.cc:407] Connection header received
0929 22:54:34.184094 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 22:54:34.184095 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 22:54:34.184111 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 22:54:34.184171 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0929 22:54:34.184744 (+ 573us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 22:54:34.184745 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 22:54:34.184746 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 22:54:34.184750 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0929 22:54:34.184765 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 22:54:34.184770 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 22:54:34.184818 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0929 22:54:34.184874 (+ 56us) server_negotiation.cc:299] Negotiation successful
0929 22:54:34.184889 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260929 22:54:34.204418 4660 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 22:54:34.204612 4660 catalog_manager.cc:1504] Loading token signing keys...
I20260929 22:54:34.208177 4660 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9379ae80da294cf79fcb466c099f2a3c: Generated new TSK 0
I20260929 22:54:34.208388 4660 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 22:54:34.217834 14927 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhWOCLk/build/release/bin/kudu with pid 4578
2026-09-29T22:54:34Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/17 (1225 ms)
[----------] 18 tests from AuthTokenIssuingTest (21991 ms total)
[----------] 2 tests from ConnectToFollowerMasterTest
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0
Loading random data
Initializing database '/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790722213368738-14927-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 22:54:34 dist-test-slave-98mk krb5kdc[4677](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 22:54:34 dist-test-slave-98mk krb5kdc[4677](info): set up 2 sockets
Sep 29 22:54:34 dist-test-slave-98mk krb5kdc[4677](info): commencing operation
krb5kdc: starting...
W20260929 22:54:35.164057 14927 mini_kdc.cc:121] Time spent starting KDC: real 0.929s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790722213368738-14927-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790722213368738-14927-0/krb5kdc/test-user.keytab.
Sep 29 22:54:35 dist-test-slave-98mk krb5kdc[4677](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790722475, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T22:54:35Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T22:54:35Z Disabled control of system clock
WARNING: no policy specified for kudu/127.14.147.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.147.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.147.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790722213368738-14927-0/krb5kdc/kudu_127.14.147.254.keytab.
Entry for principal kudu/127.14.147.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790722213368738-14927-0/krb5kdc/kudu_127.14.147.254.keytab.
I20260929 22:54:35.282411 14927 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhWOCLk/build/release/bin/kudu
/tmp/dist-test-taskhWOCLk/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790722213368738-14927-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790722213368738-14927-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790722213368738-14927-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790722213368738-14927-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.147.254:44207
--webserver_interface=127.14.147.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790722213368738-14927-0/krb5kdc/kudu_127.14.147.254.keytab
--principal=kudu/127.14.147.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.147.212:43825
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.147.254:44207,127.14.147.253:43729,127.14.147.252:34875
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790722213368738-14927-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790722213368738-14927-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790722213368738-14927-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 22:54:35.364161 4690 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260929 22:54:35.364477 4690 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 22:54:35.364504 4690 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 22:54:35.364544 4690 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 22:54:35.366367 4690 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 22:54:35.366446 4690 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 22:54:35.366466 4690 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 22:54:35.366482 4690 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 22:54:35.366498 4690 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 22:54:35.368498 4690 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.147.212:43825
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790722213368738-14927-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790722213368738-14927-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.14.147.254:44207,127.14.147.253:43729,127.14.147.252:34875
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790722213368738-14927-0/krb5kdc/kudu_127.14.147.254.keytab
--principal=kudu/127.14.147.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.147.254:44207
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790722213368738-14927-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.14.147.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4690
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790722213368738-14927-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 21:58:54 UTC on ccycloud.quasar-xguoti.root.comops.site
build id 4681390
I20260929 22:54:35.368784 4690 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 22:54:35.369010 4690 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 22:54:35.374081 4696 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 22:54:35.374112 4695 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 22:54:35.374198 4690 server_base.cc:1034] running on GCE node
W20260929 22:54:35.374112 4698 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 22:54:35.374519 4690 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 22:54:35.374830 4690 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 22:54:35.375991 4690 hybrid_clock.cc:648] HybridClock initialized: now 1790722475375971 us; error 35 us; skew 500 ppm
Sep 29 22:54:35 dist-test-slave-98mk krb5kdc[4677](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790722475, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.147.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 22:54:35.378444 4690 init.cc:377] Logged in from keytab as kudu/127.14.147.254@KRBTEST.COM (short username kudu)
I20260929 22:54:35.378861 4690 webserver.cc:460] Webserver started at http://127.14.147.254:34381/ using document root <none> and password file <none>
I20260929 22:54:35.379091 4690 fs_manager.cc:359] Metadata directory not provided
I20260929 22:54:35.379135 4690 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 22:54:35.379261 4690 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 22:54:35.380383 4690 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790722213368738-14927-0/minicluster-data/master-0/data/instance:
uuid: "746ae7c9cbf64a518827cee4606d5425"
format_stamp: "Formatted at 2026-09-29 22:54:35 on dist-test-slave-98mk"
server_key: "87b4871d4fb27c2482c39b288cf750d4"
server_key_iv: "a98ad7eeecf70b50dd7a350db09657b8"
server_key_version: "encryptionkey@0"
I20260929 22:54:35.380708 4690 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790722213368738-14927-0/minicluster-data/master-0/wal/instance:
uuid: "746ae7c9cbf64a518827cee4606d5425"
format_stamp: "Formatted at 2026-09-29 22:54:35 on dist-test-slave-98mk"
server_key: "87b4871d4fb27c2482c39b288cf750d4"
server_key_iv: "a98ad7eeecf70b50dd7a350db09657b8"
server_key_version: "encryptionkey@0"
I20260929 22:54:35.381994 4690 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260929 22:54:35.382735 4705 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 22:54:35.382920 4690 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260929 22:54:35.383006 4690 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790722213368738-14927-0/minicluster-data/master-0/data,/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790722213368738-14927-0/minicluster-data/master-0/wal
uuid: "746ae7c9cbf64a518827cee4606d5425"
format_stamp: "Formatted at 2026-09-29 22:54:35 on dist-test-slave-98mk"
server_key: "87b4871d4fb27c2482c39b288cf750d4"
server_key_iv: "a98ad7eeecf70b50dd7a350db09657b8"
server_key_version: "encryptionkey@0"
I20260929 22:54:35.383085 4690 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790722213368738-14927-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790722213368738-14927-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790722213368738-14927-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 22:54:35.419704 4690 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 22:54:35.421578 4690 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 22:54:35.421708 4690 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 22:54:35.425640 4690 rpc_server.cc:307] RPC server started. Bound to: 127.14.147.254:44207
I20260929 22:54:35.425750 4757 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.147.254:44207 every 8 connection(s)
I20260929 22:54:35.426025 4690 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790722213368738-14927-0/minicluster-data/master-0/data/info.pb
I20260929 22:54:35.426899 4758 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 22:54:35.427666 14927 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhWOCLk/build/release/bin/kudu as pid 4690
I20260929 22:54:35.427754 14927 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790722213368738-14927-0/minicluster-data/master-0/wal/instance
I20260929 22:54:35.427903 14927 external_mini_cluster.cc:1428] Setting key ad9ead376598560ea8e9b102a6dd7afe
I20260929 22:54:35.429186 4758 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.14.147.254" port: 44207 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.14.147.253@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.147.253@KRBTEST.COM" created.
Sep 29 22:54:35 dist-test-slave-98mk krb5kdc[4677](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790722475, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.147.254@KRBTEST.COM for kudu/127.14.147.254@KRBTEST.COM
I20260929 22:54:35.442430 4762 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:54:35.430063 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47556 (local address 127.14.147.254:44207)
0929 22:54:35.430242 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0929 22:54:35.430250 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 22:54:35.430389 (+ 139us) server_negotiation.cc:407] Connection header received
0929 22:54:35.431004 (+ 615us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 22:54:35.431007 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 22:54:35.431031 (+ 24us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 22:54:35.431107 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0929 22:54:35.431887 (+ 780us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:54:35.432849 (+ 962us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:54:35.434123 (+ 1274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:54:35.434509 (+ 386us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:54:35.434600 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 22:54:35.439044 (+ 4444us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 22:54:35.439054 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 22:54:35.439055 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 22:54:35.439064 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0929 22:54:35.440694 (+ 1630us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 22:54:35.441144 (+ 450us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 22:54:35.441148 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 22:54:35.441148 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 22:54:35.441241 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 22:54:35.441504 (+ 263us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 22:54:35.441504 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 22:54:35.441505 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 22:54:35.441750 (+ 245us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 22:54:35.441816 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0929 22:54:35.442070 (+ 254us) server_negotiation.cc:299] Negotiation successful
0929 22:54:35.442214 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":157,"thread_start_us":66,"threads_started":1}
I20260929 22:54:35.442448 4761 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:54:35.430016 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.147.254:44207 (local address 127.0.0.1:47556)
0929 22:54:35.430196 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0929 22:54:35.430242 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0929 22:54:35.430848 (+ 606us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 22:54:35.431137 (+ 289us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 22:54:35.431140 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 22:54:35.431535 (+ 395us) client_negotiation.cc:189] Negotiated authn=SASL
0929 22:54:35.431772 (+ 237us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 22:54:35.431778 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 22:54:35.432949 (+ 1171us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 22:54:35.432950 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 22:54:35.434022 (+ 1072us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 22:54:35.434026 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 22:54:35.434612 (+ 586us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 22:54:35.434614 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 22:54:35.434695 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 22:54:35.435219 (+ 524us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 22:54:35.435229 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0929 22:54:35.438933 (+ 3704us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 22:54:35.440783 (+ 1850us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 22:54:35.440791 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 22:54:35.440793 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 22:54:35.441027 (+ 234us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 22:54:35.441309 (+ 282us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 22:54:35.441309 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 22:54:35.441310 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 22:54:35.441417 (+ 107us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 22:54:35.441815 (+ 398us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 22:54:35.441821 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 22:54:35.441903 (+ 82us) client_negotiation.cc:769] Sending connection context
0929 22:54:35.442006 (+ 103us) client_negotiation.cc:240] Negotiation successful
0929 22:54:35.442311 (+ 305us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":125,"mutex_wait_us":119,"thread_start_us":82,"threads_started":1}
I20260929 22:54:35.443574 4758 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.14.147.253" port: 43729 } has no permanent_uuid. Determining permanent_uuid...
I20260929 22:54:35.443912 4761 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:54:35.443804 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.147.253:43729 (local address 127.0.0.1:55732)
0929 22:54:35.443848 (+ 44us) negotiation.cc:107] Waiting for socket to connect
0929 22:54:35.443889 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.147.253:43729: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":27}
W20260929 22:54:35.444037 4706 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.14.147.253:43729: connect: Connection refused (error 111)
I20260929 22:54:35.444410 4761 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:54:35.444332 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.147.253:43729 (local address 127.0.0.1:55742)
0929 22:54:35.444369 (+ 37us) negotiation.cc:107] Waiting for socket to connect
0929 22:54:35.444392 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.147.253:43729: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":24}
W20260929 22:54:35.444504 4758 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.14.147.253:43729: Network error: Client connection negotiation failed: client connection to 127.14.147.253:43729: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.147.253 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790722213368738-14927-0/krb5kdc/kudu_127.14.147.253.keytab.
Entry for principal kudu/127.14.147.253 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790722213368738-14927-0/krb5kdc/kudu_127.14.147.253.keytab.
I20260929 22:54:35.454360 14927 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhWOCLk/build/release/bin/kudu
/tmp/dist-test-taskhWOCLk/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790722213368738-14927-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790722213368738-14927-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790722213368738-14927-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790722213368738-14927-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.147.253:43729
--webserver_interface=127.14.147.253
--webserver_port=0
--keytab_file=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790722213368738-14927-0/krb5kdc/kudu_127.14.147.253.keytab
--principal=kudu/127.14.147.253
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.147.212:43825
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.147.254:44207,127.14.147.253:43729,127.14.147.252:34875
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790722213368738-14927-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790722213368738-14927-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790722213368738-14927-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 22:54:35.493747 4758 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.14.147.253" port: 43729 } attempt: 1
I20260929 22:54:35.494302 4761 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:54:35.494159 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.147.253:43729 (local address 127.0.0.1:55746)
0929 22:54:35.494219 (+ 60us) negotiation.cc:107] Waiting for socket to connect
0929 22:54:35.494268 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.147.253:43729: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
I20260929 22:54:35.494623 4761 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:54:35.494533 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.147.253:43729 (local address 127.0.0.1:55762)
0929 22:54:35.494574 (+ 41us) negotiation.cc:107] Waiting for socket to connect
0929 22:54:35.494602 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.147.253:43729: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":28}
W20260929 22:54:35.494729 4758 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.14.147.253:43729: Network error: Client connection negotiation failed: client connection to 127.14.147.253:43729: connect: Connection refused (error 111)
W20260929 22:54:35.538185 4765 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260929 22:54:35.538419 4765 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 22:54:35.538435 4765 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 22:54:35.538455 4765 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 22:54:35.539700 4765 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 22:54:35.539740 4765 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 22:54:35.539753 4765 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 22:54:35.539765 4765 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 22:54:35.539775 4765 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 22:54:35.541261 4765 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.147.212:43825
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790722213368738-14927-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790722213368738-14927-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.14.147.254:44207,127.14.147.253:43729,127.14.147.252:34875
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790722213368738-14927-0/krb5kdc/kudu_127.14.147.253.keytab
--principal=kudu/127.14.147.253
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.147.253:43729
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790722213368738-14927-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.14.147.253
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4765
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790722213368738-14927-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 21:58:54 UTC on ccycloud.quasar-xguoti.root.comops.site
build id 4681390
I20260929 22:54:35.541498 4765 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 22:54:35.541711 4765 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 22:54:35.546371 4773 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 22:54:35.546391 4771 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 22:54:35.546387 4770 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 22:54:35.546602 4765 server_base.cc:1034] running on GCE node
I20260929 22:54:35.546865 4765 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 22:54:35.547155 4765 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 22:54:35.548295 4765 hybrid_clock.cc:648] HybridClock initialized: now 1790722475548284 us; error 32 us; skew 500 ppm
Sep 29 22:54:35 dist-test-slave-98mk krb5kdc[4677](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790722475, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.147.253@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 22:54:35.550805 4765 init.cc:377] Logged in from keytab as kudu/127.14.147.253@KRBTEST.COM (short username kudu)
I20260929 22:54:35.551236 4765 webserver.cc:460] Webserver started at http://127.14.147.253:42459/ using document root <none> and password file <none>
I20260929 22:54:35.551487 4765 fs_manager.cc:359] Metadata directory not provided
I20260929 22:54:35.551549 4765 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 22:54:35.551680 4765 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 22:54:35.552747 4765 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790722213368738-14927-0/minicluster-data/master-1/data/instance:
uuid: "c1c8536b50cc416ab00186085ca75959"
format_stamp: "Formatted at 2026-09-29 22:54:35 on dist-test-slave-98mk"
server_key: "b66e84aa6ca8de4f9b62b14af3369c62"
server_key_iv: "6fbb8f2048e7792dddb283e7d455c735"
server_key_version: "encryptionkey@0"
I20260929 22:54:35.553081 4765 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790722213368738-14927-0/minicluster-data/master-1/wal/instance:
uuid: "c1c8536b50cc416ab00186085ca75959"
format_stamp: "Formatted at 2026-09-29 22:54:35 on dist-test-slave-98mk"
server_key: "b66e84aa6ca8de4f9b62b14af3369c62"
server_key_iv: "6fbb8f2048e7792dddb283e7d455c735"
server_key_version: "encryptionkey@0"
I20260929 22:54:35.554342 4765 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260929 22:54:35.555001 4780 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 22:54:35.555166 4765 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260929 22:54:35.555245 4765 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790722213368738-14927-0/minicluster-data/master-1/data,/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790722213368738-14927-0/minicluster-data/master-1/wal
uuid: "c1c8536b50cc416ab00186085ca75959"
format_stamp: "Formatted at 2026-09-29 22:54:35 on dist-test-slave-98mk"
server_key: "b66e84aa6ca8de4f9b62b14af3369c62"
server_key_iv: "6fbb8f2048e7792dddb283e7d455c735"
server_key_version: "encryptionkey@0"
I20260929 22:54:35.555312 4765 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790722213368738-14927-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790722213368738-14927-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790722213368738-14927-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 22:54:35.562896 4758 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.14.147.253" port: 43729 } attempt: 2
I20260929 22:54:35.563395 4761 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:54:35.563253 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.147.253:43729 (local address 127.0.0.1:55778)
0929 22:54:35.563326 (+ 73us) negotiation.cc:107] Waiting for socket to connect
0929 22:54:35.563359 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.147.253:43729: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
I20260929 22:54:35.563741 4761 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:54:35.563657 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.147.253:43729 (local address 127.0.0.1:55792)
0929 22:54:35.563699 (+ 42us) negotiation.cc:107] Waiting for socket to connect
0929 22:54:35.563724 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.147.253:43729: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":27}
W20260929 22:54:35.563835 4758 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.14.147.253:43729: Network error: Client connection negotiation failed: client connection to 127.14.147.253:43729: connect: Connection refused (error 111)
I20260929 22:54:35.566965 4765 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 22:54:35.568984 4765 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 22:54:35.569123 4765 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 22:54:35.573107 4765 rpc_server.cc:307] RPC server started. Bound to: 127.14.147.253:43729
I20260929 22:54:35.573137 4832 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.147.253:43729 every 8 connection(s)
I20260929 22:54:35.573468 4765 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790722213368738-14927-0/minicluster-data/master-1/data/info.pb
I20260929 22:54:35.574616 4833 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 22:54:35.576215 4833 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.14.147.254" port: 44207 } has no permanent_uuid. Determining permanent_uuid...
I20260929 22:54:35.579320 14927 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhWOCLk/build/release/bin/kudu as pid 4765
I20260929 22:54:35.579454 14927 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790722213368738-14927-0/minicluster-data/master-1/wal/instance
I20260929 22:54:35.579627 14927 external_mini_cluster.cc:1428] Setting key 9c44ae804682f465b1489b60d91cb648
Sep 29 22:54:35 dist-test-slave-98mk krb5kdc[4677](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790722475, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.147.253@KRBTEST.COM for kudu/127.14.147.254@KRBTEST.COM
I20260929 22:54:35.585438 4762 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:54:35.576970 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47564 (local address 127.14.147.254:44207)
0929 22:54:35.577044 (+ 74us) server_negotiation.cc:206] Beginning negotiation
0929 22:54:35.577046 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 22:54:35.577222 (+ 176us) server_negotiation.cc:407] Connection header received
0929 22:54:35.577711 (+ 489us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 22:54:35.577712 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 22:54:35.577723 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 22:54:35.577784 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0929 22:54:35.578369 (+ 585us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:54:35.578860 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:54:35.580191 (+ 1331us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:54:35.580407 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:54:35.580469 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 22:54:35.582723 (+ 2254us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 22:54:35.582724 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 22:54:35.582725 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 22:54:35.582732 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0929 22:54:35.584010 (+ 1278us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 22:54:35.584501 (+ 491us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 22:54:35.584502 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 22:54:35.584502 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 22:54:35.584549 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 22:54:35.584806 (+ 257us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 22:54:35.584806 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 22:54:35.584807 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 22:54:35.584907 (+ 100us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 22:54:35.584949 (+ 42us) server_negotiation.cc:1035] Waiting for connection context
0929 22:54:35.585304 (+ 355us) server_negotiation.cc:299] Negotiation successful
0929 22:54:35.585369 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260929 22:54:35.585726 4835 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:54:35.576912 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.147.254:44207 (local address 127.0.0.1:47564)
0929 22:54:35.577110 (+ 198us) negotiation.cc:107] Waiting for socket to connect
0929 22:54:35.577153 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0929 22:54:35.577636 (+ 483us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 22:54:35.577837 (+ 201us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 22:54:35.577841 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 22:54:35.578104 (+ 263us) client_negotiation.cc:189] Negotiated authn=SASL
0929 22:54:35.578301 (+ 197us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 22:54:35.578305 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 22:54:35.578926 (+ 621us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 22:54:35.578927 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 22:54:35.580099 (+ 1172us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 22:54:35.580104 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 22:54:35.580487 (+ 383us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 22:54:35.580488 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 22:54:35.580640 (+ 152us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 22:54:35.581234 (+ 594us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 22:54:35.581243 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0929 22:54:35.582626 (+ 1383us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 22:54:35.584095 (+ 1469us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 22:54:35.584105 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 22:54:35.584108 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 22:54:35.584426 (+ 318us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 22:54:35.584601 (+ 175us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 22:54:35.584602 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 22:54:35.584602 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 22:54:35.584747 (+ 145us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 22:54:35.584960 (+ 213us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 22:54:35.584966 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 22:54:35.585177 (+ 211us) client_negotiation.cc:769] Sending connection context
0929 22:54:35.585297 (+ 120us) client_negotiation.cc:240] Negotiation successful
0929 22:54:35.585474 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":154,"thread_start_us":101,"threads_started":1}
I20260929 22:54:35.586253 4833 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.14.147.253" port: 43729 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.14.147.252@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.147.252@KRBTEST.COM" created.
Sep 29 22:54:35 dist-test-slave-98mk krb5kdc[4677](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790722475, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.147.253@KRBTEST.COM for kudu/127.14.147.253@KRBTEST.COM
I20260929 22:54:35.593669 4835 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:54:35.586499 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.147.253:43729 (local address 127.0.0.1:55804)
0929 22:54:35.586550 (+ 51us) negotiation.cc:107] Waiting for socket to connect
0929 22:54:35.586561 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0929 22:54:35.586606 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 22:54:35.586836 (+ 230us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 22:54:35.586837 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 22:54:35.587005 (+ 168us) client_negotiation.cc:189] Negotiated authn=SASL
0929 22:54:35.587070 (+ 65us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 22:54:35.587072 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 22:54:35.587948 (+ 876us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 22:54:35.587949 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 22:54:35.588438 (+ 489us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 22:54:35.588439 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 22:54:35.588939 (+ 500us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 22:54:35.588940 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 22:54:35.588997 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 22:54:35.589330 (+ 333us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 22:54:35.589333 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0929 22:54:35.590733 (+ 1400us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 22:54:35.592566 (+ 1833us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 22:54:35.592568 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 22:54:35.592568 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 22:54:35.592717 (+ 149us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 22:54:35.592888 (+ 171us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 22:54:35.592889 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 22:54:35.592889 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 22:54:35.592942 (+ 53us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 22:54:35.593327 (+ 385us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 22:54:35.593328 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 22:54:35.593370 (+ 42us) client_negotiation.cc:769] Sending connection context
0929 22:54:35.593436 (+ 66us) client_negotiation.cc:240] Negotiation successful
0929 22:54:35.593549 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":29}
I20260929 22:54:35.593755 4837 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:54:35.586564 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55804 (local address 127.14.147.253:43729)
0929 22:54:35.586711 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0929 22:54:35.586716 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 22:54:35.586731 (+ 15us) server_negotiation.cc:407] Connection header received
0929 22:54:35.586749 (+ 18us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 22:54:35.586750 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 22:54:35.586761 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 22:54:35.586818 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0929 22:54:35.587135 (+ 317us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:54:35.587876 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:54:35.588506 (+ 630us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:54:35.588870 (+ 364us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:54:35.588923 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 22:54:35.590811 (+ 1888us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 22:54:35.590820 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 22:54:35.590821 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 22:54:35.590828 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0929 22:54:35.592493 (+ 1665us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 22:54:35.592783 (+ 290us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 22:54:35.592786 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 22:54:35.592787 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 22:54:35.592840 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 22:54:35.592989 (+ 149us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 22:54:35.592990 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 22:54:35.592991 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 22:54:35.593234 (+ 243us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 22:54:35.593336 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0929 22:54:35.593471 (+ 135us) server_negotiation.cc:299] Negotiation successful
0929 22:54:35.593620 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":64,"server-negotiator.queue_time_us":120,"thread_start_us":87,"threads_started":1}
I20260929 22:54:35.594305 4833 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.14.147.252" port: 34875 } has no permanent_uuid. Determining permanent_uuid...
I20260929 22:54:35.594667 4835 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:54:35.594536 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.147.252:34875 (local address 127.0.0.1:38118)
0929 22:54:35.594576 (+ 40us) negotiation.cc:107] Waiting for socket to connect
0929 22:54:35.594624 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.147.252:34875: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":24}
W20260929 22:54:35.594803 4784 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.14.147.252:34875: connect: Connection refused (error 111)
I20260929 22:54:35.595495 4835 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:54:35.595380 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.147.252:34875 (local address 127.0.0.1:38120)
0929 22:54:35.595447 (+ 67us) negotiation.cc:107] Waiting for socket to connect
0929 22:54:35.595472 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.147.252:34875: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":26}
W20260929 22:54:35.595603 4833 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.14.147.252:34875: Network error: Client connection negotiation failed: client connection to 127.14.147.252:34875: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.147.252 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790722213368738-14927-0/krb5kdc/kudu_127.14.147.252.keytab.
Entry for principal kudu/127.14.147.252 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790722213368738-14927-0/krb5kdc/kudu_127.14.147.252.keytab.
I20260929 22:54:35.606251 14927 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhWOCLk/build/release/bin/kudu
/tmp/dist-test-taskhWOCLk/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790722213368738-14927-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790722213368738-14927-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790722213368738-14927-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790722213368738-14927-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.147.252:34875
--webserver_interface=127.14.147.252
--webserver_port=0
--keytab_file=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790722213368738-14927-0/krb5kdc/kudu_127.14.147.252.keytab
--principal=kudu/127.14.147.252
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.147.212:43825
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.147.254:44207,127.14.147.253:43729,127.14.147.252:34875
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790722213368738-14927-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790722213368738-14927-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790722213368738-14927-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 22:54:35.644824 4833 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.14.147.252" port: 34875 } attempt: 1
I20260929 22:54:35.645305 4835 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:54:35.645191 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.147.252:34875 (local address 127.0.0.1:38122)
0929 22:54:35.645243 (+ 52us) negotiation.cc:107] Waiting for socket to connect
0929 22:54:35.645275 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.147.252:34875: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":28}
I20260929 22:54:35.645617 4835 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:54:35.645541 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.147.252:34875 (local address 127.0.0.1:38138)
0929 22:54:35.645575 (+ 34us) negotiation.cc:107] Waiting for socket to connect
0929 22:54:35.645598 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.147.252:34875: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":21}
W20260929 22:54:35.645704 4833 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.14.147.252:34875: Network error: Client connection negotiation failed: client connection to 127.14.147.252:34875: connect: Connection refused (error 111)
I20260929 22:54:35.654999 4758 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.14.147.253" port: 43729 } attempt: 3
Sep 29 22:54:35 dist-test-slave-98mk krb5kdc[4677](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790722475, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.147.254@KRBTEST.COM for kudu/127.14.147.253@KRBTEST.COM
I20260929 22:54:35.661972 4761 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:54:35.655359 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.147.253:43729 (local address 127.0.0.1:55814)
0929 22:54:35.655445 (+ 86us) negotiation.cc:107] Waiting for socket to connect
0929 22:54:35.655462 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0929 22:54:35.655528 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 22:54:35.655669 (+ 141us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 22:54:35.655670 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 22:54:35.655859 (+ 189us) client_negotiation.cc:189] Negotiated authn=SASL
0929 22:54:35.655938 (+ 79us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 22:54:35.655940 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 22:54:35.656599 (+ 659us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 22:54:35.656600 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 22:54:35.657436 (+ 836us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 22:54:35.657439 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 22:54:35.657808 (+ 369us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 22:54:35.657809 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 22:54:35.657881 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 22:54:35.658304 (+ 423us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 22:54:35.658310 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0929 22:54:35.659650 (+ 1340us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 22:54:35.661057 (+ 1407us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 22:54:35.661058 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 22:54:35.661059 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 22:54:35.661220 (+ 161us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 22:54:35.661384 (+ 164us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 22:54:35.661385 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 22:54:35.661385 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 22:54:35.661440 (+ 55us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 22:54:35.661660 (+ 220us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 22:54:35.661660 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 22:54:35.661708 (+ 48us) client_negotiation.cc:769] Sending connection context
0929 22:54:35.661768 (+ 60us) client_negotiation.cc:240] Negotiation successful
0929 22:54:35.661871 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":38}
I20260929 22:54:35.661972 4837 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:54:35.655448 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55814 (local address 127.14.147.253:43729)
0929 22:54:35.655500 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0929 22:54:35.655502 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 22:54:35.655512 (+ 10us) server_negotiation.cc:407] Connection header received
0929 22:54:35.655590 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 22:54:35.655591 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 22:54:35.655600 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 22:54:35.655647 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0929 22:54:35.656004 (+ 357us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:54:35.656522 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:54:35.657512 (+ 990us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:54:35.657740 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:54:35.657802 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 22:54:35.659737 (+ 1935us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 22:54:35.659739 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 22:54:35.659739 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 22:54:35.659746 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0929 22:54:35.660982 (+ 1236us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 22:54:35.661278 (+ 296us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 22:54:35.661280 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 22:54:35.661280 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 22:54:35.661332 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 22:54:35.661495 (+ 163us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 22:54:35.661495 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 22:54:35.661496 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 22:54:35.661606 (+ 110us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 22:54:35.661660 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0929 22:54:35.661796 (+ 136us) server_negotiation.cc:299] Negotiation successful
0929 22:54:35.661870 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260929 22:54:35.662412 4758 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.14.147.252" port: 34875 } has no permanent_uuid. Determining permanent_uuid...
I20260929 22:54:35.662730 4761 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:54:35.662641 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.147.252:34875 (local address 127.0.0.1:38152)
0929 22:54:35.662683 (+ 42us) negotiation.cc:107] Waiting for socket to connect
0929 22:54:35.662711 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.147.252:34875: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":26}
I20260929 22:54:35.663033 4761 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:54:35.662951 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.147.252:34875 (local address 127.0.0.1:38164)
0929 22:54:35.662988 (+ 37us) negotiation.cc:107] Waiting for socket to connect
0929 22:54:35.663011 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.147.252:34875: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":24}
W20260929 22:54:35.663116 4758 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.14.147.252:34875: Network error: Client connection negotiation failed: client connection to 127.14.147.252:34875: connect: Connection refused (error 111)
W20260929 22:54:35.688632 4840 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260929 22:54:35.688872 4840 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 22:54:35.688896 4840 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 22:54:35.688925 4840 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 22:54:35.690176 4840 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 22:54:35.690227 4840 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 22:54:35.690246 4840 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 22:54:35.690263 4840 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 22:54:35.690279 4840 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 22:54:35.691900 4840 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.147.212:43825
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790722213368738-14927-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790722213368738-14927-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.14.147.254:44207,127.14.147.253:43729,127.14.147.252:34875
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790722213368738-14927-0/krb5kdc/kudu_127.14.147.252.keytab
--principal=kudu/127.14.147.252
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.147.252:34875
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790722213368738-14927-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.14.147.252
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4840
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790722213368738-14927-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 21:58:54 UTC on ccycloud.quasar-xguoti.root.comops.site
build id 4681390
I20260929 22:54:35.692162 4840 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 22:54:35.692381 4840 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260929 22:54:35.694258 4758 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.14.147.252" port: 34875 } attempt: 1
I20260929 22:54:35.694682 4761 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:54:35.694560 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.147.252:34875 (local address 127.0.0.1:38170)
0929 22:54:35.694622 (+ 62us) negotiation.cc:107] Waiting for socket to connect
0929 22:54:35.694654 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.147.252:34875: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
I20260929 22:54:35.694962 4761 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:54:35.694887 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.147.252:34875 (local address 127.0.0.1:38174)
0929 22:54:35.694921 (+ 34us) negotiation.cc:107] Waiting for socket to connect
0929 22:54:35.694944 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.147.252:34875: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":22}
W20260929 22:54:35.695035 4758 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.14.147.252:34875: Network error: Client connection negotiation failed: client connection to 127.14.147.252:34875: connect: Connection refused (error 111)
W20260929 22:54:35.697217 4845 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 22:54:35.697237 4846 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 22:54:35.697260 4848 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 22:54:35.697582 4840 server_base.cc:1034] running on GCE node
I20260929 22:54:35.697795 4840 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 22:54:35.698046 4840 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 22:54:35.699180 4840 hybrid_clock.cc:648] HybridClock initialized: now 1790722475699167 us; error 26 us; skew 500 ppm
Sep 29 22:54:35 dist-test-slave-98mk krb5kdc[4677](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790722475, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.147.252@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 22:54:35.701992 4840 init.cc:377] Logged in from keytab as kudu/127.14.147.252@KRBTEST.COM (short username kudu)
I20260929 22:54:35.702538 4840 webserver.cc:460] Webserver started at http://127.14.147.252:34015/ using document root <none> and password file <none>
I20260929 22:54:35.702766 4840 fs_manager.cc:359] Metadata directory not provided
I20260929 22:54:35.702813 4840 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 22:54:35.702940 4840 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 22:54:35.704084 4840 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790722213368738-14927-0/minicluster-data/master-2/data/instance:
uuid: "c5419fdb9421499aaedbe20fa12d63e2"
format_stamp: "Formatted at 2026-09-29 22:54:35 on dist-test-slave-98mk"
server_key: "312e4f6d1fdbb69822705501949f9bf8"
server_key_iv: "edd45cab3fd4f6a748a8b71ee18fd5b1"
server_key_version: "encryptionkey@0"
I20260929 22:54:35.704416 4840 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790722213368738-14927-0/minicluster-data/master-2/wal/instance:
uuid: "c5419fdb9421499aaedbe20fa12d63e2"
format_stamp: "Formatted at 2026-09-29 22:54:35 on dist-test-slave-98mk"
server_key: "312e4f6d1fdbb69822705501949f9bf8"
server_key_iv: "edd45cab3fd4f6a748a8b71ee18fd5b1"
server_key_version: "encryptionkey@0"
I20260929 22:54:35.705729 4840 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260929 22:54:35.706497 4855 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 22:54:35.706686 4840 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260929 22:54:35.706779 4840 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790722213368738-14927-0/minicluster-data/master-2/data,/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790722213368738-14927-0/minicluster-data/master-2/wal
uuid: "c5419fdb9421499aaedbe20fa12d63e2"
format_stamp: "Formatted at 2026-09-29 22:54:35 on dist-test-slave-98mk"
server_key: "312e4f6d1fdbb69822705501949f9bf8"
server_key_iv: "edd45cab3fd4f6a748a8b71ee18fd5b1"
server_key_version: "encryptionkey@0"
I20260929 22:54:35.706851 4840 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790722213368738-14927-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790722213368738-14927-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790722213368738-14927-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 22:54:35.713845 4833 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.14.147.252" port: 34875 } attempt: 2
I20260929 22:54:35.714340 4835 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:54:35.714206 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.147.252:34875 (local address 127.0.0.1:38178)
0929 22:54:35.714261 (+ 55us) negotiation.cc:107] Waiting for socket to connect
0929 22:54:35.714306 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.147.252:34875: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
I20260929 22:54:35.714704 4835 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:54:35.714629 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.147.252:34875 (local address 127.0.0.1:38186)
0929 22:54:35.714664 (+ 35us) negotiation.cc:107] Waiting for socket to connect
0929 22:54:35.714686 (+ 22us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.147.252:34875: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":24}
W20260929 22:54:35.714807 4833 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.14.147.252:34875: Network error: Client connection negotiation failed: client connection to 127.14.147.252:34875: connect: Connection refused (error 111)
I20260929 22:54:35.723754 4840 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 22:54:35.725610 4840 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 22:54:35.725728 4840 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 22:54:35.729494 4840 rpc_server.cc:307] RPC server started. Bound to: 127.14.147.252:34875
I20260929 22:54:35.729540 4907 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.147.252:34875 every 8 connection(s)
I20260929 22:54:35.729859 4840 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790722213368738-14927-0/minicluster-data/master-2/data/info.pb
I20260929 22:54:35.730815 4908 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 22:54:35.731256 14927 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhWOCLk/build/release/bin/kudu as pid 4840
I20260929 22:54:35.731351 14927 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790722213368738-14927-0/minicluster-data/master-2/wal/instance
I20260929 22:54:35.731571 14927 external_mini_cluster.cc:1428] Setting key 1b04654735f19cb2085a7f2bbeb5b1d2
I20260929 22:54:35.732604 4908 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.14.147.254" port: 44207 } has no permanent_uuid. Determining permanent_uuid...
Sep 29 22:54:35 dist-test-slave-98mk krb5kdc[4677](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790722475, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.147.252@KRBTEST.COM for kudu/127.14.147.254@KRBTEST.COM
Sep 29 22:54:35 dist-test-slave-98mk krb5kdc[4677](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790722475, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.147.252@KRBTEST.COM
I20260929 22:54:35.740731 4762 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:54:35.733299 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47568 (local address 127.14.147.254:44207)
0929 22:54:35.733347 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0929 22:54:35.733349 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 22:54:35.733448 (+ 99us) server_negotiation.cc:407] Connection header received
0929 22:54:35.733530 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 22:54:35.733531 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 22:54:35.733538 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 22:54:35.733579 (+ 41us) server_negotiation.cc:226] Negotiated authn=SASL
0929 22:54:35.734180 (+ 601us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:54:35.734641 (+ 461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:54:35.735487 (+ 846us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:54:35.735667 (+ 180us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:54:35.735717 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 22:54:35.737961 (+ 2244us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 22:54:35.737963 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 22:54:35.737963 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 22:54:35.737971 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0929 22:54:35.739219 (+ 1248us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 22:54:35.739671 (+ 452us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 22:54:35.739672 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 22:54:35.739672 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 22:54:35.739723 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 22:54:35.739966 (+ 243us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 22:54:35.739967 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 22:54:35.739967 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 22:54:35.740102 (+ 135us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 22:54:35.740174 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0929 22:54:35.740561 (+ 387us) server_negotiation.cc:299] Negotiation successful
0929 22:54:35.740661 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260929 22:54:35.740869 4912 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:54:35.733206 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.147.254:44207 (local address 127.0.0.1:47568)
0929 22:54:35.733351 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0929 22:54:35.733393 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0929 22:54:35.733466 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 22:54:35.733599 (+ 133us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 22:54:35.733602 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 22:54:35.733936 (+ 334us) client_negotiation.cc:189] Negotiated authn=SASL
0929 22:54:35.734046 (+ 110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 22:54:35.734052 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 22:54:35.734719 (+ 667us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 22:54:35.734720 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 22:54:35.735376 (+ 656us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 22:54:35.735378 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 22:54:35.735729 (+ 351us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 22:54:35.735731 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 22:54:35.735838 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 22:54:35.736331 (+ 493us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 22:54:35.736341 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0929 22:54:35.737872 (+ 1531us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 22:54:35.739290 (+ 1418us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 22:54:35.739295 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 22:54:35.739297 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 22:54:35.739604 (+ 307us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 22:54:35.739775 (+ 171us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 22:54:35.739777 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 22:54:35.739777 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 22:54:35.739902 (+ 125us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 22:54:35.740218 (+ 316us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 22:54:35.740224 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 22:54:35.740406 (+ 182us) client_negotiation.cc:769] Sending connection context
0929 22:54:35.740507 (+ 101us) client_negotiation.cc:240] Negotiation successful
0929 22:54:35.740640 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":84,"threads_started":1}
I20260929 22:54:35.741402 4908 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.14.147.253" port: 43729 } has no permanent_uuid. Determining permanent_uuid...
I20260929 22:54:35.741492 4911 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:54:35.732096 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38194 (local address 127.14.147.252:34875)
0929 22:54:35.732251 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0929 22:54:35.732258 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 22:54:35.732282 (+ 24us) server_negotiation.cc:407] Connection header received
0929 22:54:35.732689 (+ 407us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 22:54:35.732696 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 22:54:35.732741 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 22:54:35.732852 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0929 22:54:35.733596 (+ 744us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:54:35.734675 (+ 1079us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:54:35.735404 (+ 729us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:54:35.735803 (+ 399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:54:35.735864 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 22:54:35.738291 (+ 2427us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 22:54:35.738299 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 22:54:35.738300 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 22:54:35.738311 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0929 22:54:35.739926 (+ 1615us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 22:54:35.740287 (+ 361us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 22:54:35.740290 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 22:54:35.740290 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 22:54:35.740360 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 22:54:35.740627 (+ 267us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 22:54:35.740628 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 22:54:35.740628 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 22:54:35.740782 (+ 154us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 22:54:35.740931 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0929 22:54:35.741312 (+ 381us) server_negotiation.cc:299] Negotiation successful
0929 22:54:35.741386 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":110,"thread_start_us":66,"threads_started":1}
Sep 29 22:54:35 dist-test-slave-98mk krb5kdc[4677](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790722475, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.147.252@KRBTEST.COM for kudu/127.14.147.253@KRBTEST.COM
Sep 29 22:54:35 dist-test-slave-98mk krb5kdc[4677](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790722475, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.147.254@KRBTEST.COM
I20260929 22:54:35.747592 4912 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:54:35.741670 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.147.253:43729 (local address 127.0.0.1:55820)
0929 22:54:35.741708 (+ 38us) negotiation.cc:107] Waiting for socket to connect
0929 22:54:35.741717 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0929 22:54:35.741748 (+ 31us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 22:54:35.741883 (+ 135us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 22:54:35.741884 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 22:54:35.742026 (+ 142us) client_negotiation.cc:189] Negotiated authn=SASL
0929 22:54:35.742076 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 22:54:35.742077 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 22:54:35.742730 (+ 653us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 22:54:35.742731 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 22:54:35.743212 (+ 481us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 22:54:35.743214 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 22:54:35.743608 (+ 394us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 22:54:35.743609 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 22:54:35.743677 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 22:54:35.744072 (+ 395us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 22:54:35.744084 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0929 22:54:35.745262 (+ 1178us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 22:54:35.746683 (+ 1421us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 22:54:35.746683 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 22:54:35.746684 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 22:54:35.746856 (+ 172us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 22:54:35.747045 (+ 189us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 22:54:35.747045 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 22:54:35.747046 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 22:54:35.747106 (+ 60us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 22:54:35.747358 (+ 252us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 22:54:35.747358 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 22:54:35.747389 (+ 31us) client_negotiation.cc:769] Sending connection context
0929 22:54:35.747456 (+ 67us) client_negotiation.cc:240] Negotiation successful
0929 22:54:35.747530 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":25}
I20260929 22:54:35.747643 4837 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:54:35.741738 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55820 (local address 127.14.147.253:43729)
0929 22:54:35.741779 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0929 22:54:35.741781 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 22:54:35.741789 (+ 8us) server_negotiation.cc:407] Connection header received
0929 22:54:35.741811 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 22:54:35.741812 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 22:54:35.741823 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 22:54:35.741871 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0929 22:54:35.742162 (+ 291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:54:35.742659 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:54:35.743281 (+ 622us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:54:35.743549 (+ 268us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:54:35.743606 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 22:54:35.745339 (+ 1733us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 22:54:35.745341 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 22:54:35.745341 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 22:54:35.745349 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0929 22:54:35.746617 (+ 1268us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 22:54:35.746928 (+ 311us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 22:54:35.746929 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 22:54:35.746929 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 22:54:35.746984 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 22:54:35.747169 (+ 185us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 22:54:35.747170 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 22:54:35.747170 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 22:54:35.747302 (+ 132us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 22:54:35.747367 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0929 22:54:35.747481 (+ 114us) server_negotiation.cc:299] Negotiation successful
0929 22:54:35.747551 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260929 22:54:35.747959 4908 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.14.147.252" port: 34875 } has no permanent_uuid. Determining permanent_uuid...
I20260929 22:54:35.748775 4762 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:54:35.742225 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47570 (local address 127.14.147.254:44207)
0929 22:54:35.742261 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0929 22:54:35.742263 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 22:54:35.742269 (+ 6us) server_negotiation.cc:407] Connection header received
0929 22:54:35.742286 (+ 17us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 22:54:35.742287 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 22:54:35.742294 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 22:54:35.742336 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0929 22:54:35.742750 (+ 414us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:54:35.743288 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:54:35.744025 (+ 737us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:54:35.744289 (+ 264us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:54:35.744357 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 22:54:35.746350 (+ 1993us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 22:54:35.746351 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 22:54:35.746352 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 22:54:35.746360 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0929 22:54:35.747741 (+ 1381us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 22:54:35.748082 (+ 341us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 22:54:35.748083 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 22:54:35.748083 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 22:54:35.748137 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 22:54:35.748323 (+ 186us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 22:54:35.748324 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 22:54:35.748324 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 22:54:35.748423 (+ 99us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 22:54:35.748583 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0929 22:54:35.748619 (+ 36us) server_negotiation.cc:299] Negotiation successful
0929 22:54:35.748687 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
Sep 29 22:54:35 dist-test-slave-98mk krb5kdc[4677](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790722475, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.147.252@KRBTEST.COM for kudu/127.14.147.252@KRBTEST.COM
Sep 29 22:54:35 dist-test-slave-98mk krb5kdc[4677](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790722475, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.147.253@KRBTEST.COM
I20260929 22:54:35.753978 4912 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:54:35.748186 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.147.252:34875 (local address 127.0.0.1:38198)
0929 22:54:35.748229 (+ 43us) negotiation.cc:107] Waiting for socket to connect
0929 22:54:35.748240 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0929 22:54:35.748290 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 22:54:35.748398 (+ 108us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 22:54:35.748399 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 22:54:35.748511 (+ 112us) client_negotiation.cc:189] Negotiated authn=SASL
0929 22:54:35.748535 (+ 24us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 22:54:35.748536 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 22:54:35.749321 (+ 785us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 22:54:35.749322 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 22:54:35.749853 (+ 531us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 22:54:35.749855 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 22:54:35.750163 (+ 308us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 22:54:35.750164 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 22:54:35.750214 (+ 50us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 22:54:35.750540 (+ 326us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 22:54:35.750544 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0929 22:54:35.751698 (+ 1154us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 22:54:35.753015 (+ 1317us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 22:54:35.753015 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 22:54:35.753016 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 22:54:35.753162 (+ 146us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 22:54:35.753344 (+ 182us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 22:54:35.753345 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 22:54:35.753345 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 22:54:35.753398 (+ 53us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 22:54:35.753640 (+ 242us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 22:54:35.753640 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 22:54:35.753683 (+ 43us) client_negotiation.cc:769] Sending connection context
0929 22:54:35.753747 (+ 64us) client_negotiation.cc:240] Negotiation successful
0929 22:54:35.753839 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":28}
I20260929 22:54:35.753984 4911 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:54:35.748219 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38198 (local address 127.14.147.252:34875)
0929 22:54:35.748253 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0929 22:54:35.748254 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0929 22:54:35.748277 (+ 23us) server_negotiation.cc:407] Connection header received
0929 22:54:35.748352 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 22:54:35.748353 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 22:54:35.748359 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 22:54:35.748396 (+ 37us) server_negotiation.cc:226] Negotiated authn=SASL
0929 22:54:35.748591 (+ 195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:54:35.749153 (+ 562us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:54:35.749919 (+ 766us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:54:35.750106 (+ 187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:54:35.750154 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 22:54:35.751779 (+ 1625us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 22:54:35.751780 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 22:54:35.751780 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 22:54:35.751787 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0929 22:54:35.752947 (+ 1160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 22:54:35.753225 (+ 278us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 22:54:35.753226 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 22:54:35.753227 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 22:54:35.753281 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 22:54:35.753458 (+ 177us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 22:54:35.753458 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 22:54:35.753459 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 22:54:35.753580 (+ 121us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 22:54:35.753642 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0929 22:54:35.753777 (+ 135us) server_negotiation.cc:299] Negotiation successful
0929 22:54:35.753882 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":48,"server-negotiator.queue_time_us":16}
I20260929 22:54:35.755345 4908 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c5419fdb9421499aaedbe20fa12d63e2: Bootstrap starting.
I20260929 22:54:35.755689 4837 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:54:35.749325 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55826 (local address 127.14.147.253:43729)
0929 22:54:35.749356 (+ 31us) server_negotiation.cc:206] Beginning negotiation
0929 22:54:35.749358 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 22:54:35.749364 (+ 6us) server_negotiation.cc:407] Connection header received
0929 22:54:35.749381 (+ 17us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 22:54:35.749381 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 22:54:35.749389 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 22:54:35.749445 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0929 22:54:35.749811 (+ 366us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:54:35.750346 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:54:35.751027 (+ 681us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:54:35.751216 (+ 189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:54:35.751264 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 22:54:35.753295 (+ 2031us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 22:54:35.753296 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 22:54:35.753296 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 22:54:35.753302 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0929 22:54:35.754701 (+ 1399us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 22:54:35.754983 (+ 282us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 22:54:35.754984 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 22:54:35.754984 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 22:54:35.755034 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 22:54:35.755194 (+ 160us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 22:54:35.755195 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 22:54:35.755195 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 22:54:35.755299 (+ 104us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 22:54:35.755493 (+ 194us) server_negotiation.cc:1035] Waiting for connection context
0929 22:54:35.755529 (+ 36us) server_negotiation.cc:299] Negotiation successful
0929 22:54:35.755604 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260929 22:54:35.756304 4908 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c5419fdb9421499aaedbe20fa12d63e2: Neither blocks nor log segments found. Creating new log.
I20260929 22:54:35.756606 4908 log.cc:824] T 00000000000000000000000000000000 P c5419fdb9421499aaedbe20fa12d63e2: Log is configured to *not* fsync() on all Append() calls
I20260929 22:54:35.757288 4908 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c5419fdb9421499aaedbe20fa12d63e2: No bootstrap required, opened a new log
I20260929 22:54:35.757282 14927 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 22:54:35.758950 4908 raft_consensus.cc:348] T 00000000000000000000000000000000 P c5419fdb9421499aaedbe20fa12d63e2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "746ae7c9cbf64a518827cee4606d5425" member_type: VOTER last_known_addr { host: "127.14.147.254" port: 44207 } } peers { permanent_uuid: "c1c8536b50cc416ab00186085ca75959" member_type: VOTER last_known_addr { host: "127.14.147.253" port: 43729 } } peers { permanent_uuid: "c5419fdb9421499aaedbe20fa12d63e2" member_type: VOTER last_known_addr { host: "127.14.147.252" port: 34875 } }
I20260929 22:54:35.759084 4908 raft_consensus.cc:729] T 00000000000000000000000000000000 P c5419fdb9421499aaedbe20fa12d63e2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c5419fdb9421499aaedbe20fa12d63e2, State: Initialized, Role: FOLLOWER
I20260929 22:54:35.759148 4908 consensus_queue.cc:260] T 00000000000000000000000000000000 P c5419fdb9421499aaedbe20fa12d63e2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "746ae7c9cbf64a518827cee4606d5425" member_type: VOTER last_known_addr { host: "127.14.147.254" port: 44207 } } peers { permanent_uuid: "c1c8536b50cc416ab00186085ca75959" member_type: VOTER last_known_addr { host: "127.14.147.253" port: 43729 } } peers { permanent_uuid: "c5419fdb9421499aaedbe20fa12d63e2" member_type: VOTER last_known_addr { host: "127.14.147.252" port: 34875 } }
I20260929 22:54:35.758486 4732 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "746ae7c9cbf64a518827cee4606d5425"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47570
I20260929 22:54:35.759388 4914 sys_catalog.cc:455] T 00000000000000000000000000000000 P c5419fdb9421499aaedbe20fa12d63e2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "746ae7c9cbf64a518827cee4606d5425" member_type: VOTER last_known_addr { host: "127.14.147.254" port: 44207 } } peers { permanent_uuid: "c1c8536b50cc416ab00186085ca75959" member_type: VOTER last_known_addr { host: "127.14.147.253" port: 43729 } } peers { permanent_uuid: "c5419fdb9421499aaedbe20fa12d63e2" member_type: VOTER last_known_addr { host: "127.14.147.252" port: 34875 } } }
I20260929 22:54:35.759485 4908 sys_catalog.cc:564] T 00000000000000000000000000000000 P c5419fdb9421499aaedbe20fa12d63e2 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 22:54:35.759505 4914 sys_catalog.cc:458] T 00000000000000000000000000000000 P c5419fdb9421499aaedbe20fa12d63e2 [sys.catalog]: This master's current role is: FOLLOWER
W20260929 22:54:35.761051 4925 catalog_manager.cc:1558] T 00000000000000000000000000000000 P c5419fdb9421499aaedbe20fa12d63e2: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 22:54:35.761116 4925 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260929 22:54:35.761263 4840 master_runner.cc:186] Error getting master registration for 127.14.147.254:44207: OK, instance_id { permanent_uuid: "746ae7c9cbf64a518827cee4606d5425" instance_seqno: 1790722475424513 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260929 22:54:35.763236 4840 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260929 22:54:35.770195 4758 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.14.147.252" port: 34875 } attempt: 2
Sep 29 22:54:35 dist-test-slave-98mk krb5kdc[4677](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790722475, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.147.254@KRBTEST.COM for kudu/127.14.147.252@KRBTEST.COM
I20260929 22:54:35.777776 4761 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:54:35.770729 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.147.252:34875 (local address 127.0.0.1:38208)
0929 22:54:35.770788 (+ 59us) negotiation.cc:107] Waiting for socket to connect
0929 22:54:35.770799 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0929 22:54:35.770840 (+ 41us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 22:54:35.770997 (+ 157us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 22:54:35.770997 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 22:54:35.771185 (+ 188us) client_negotiation.cc:189] Negotiated authn=SASL
0929 22:54:35.771240 (+ 55us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 22:54:35.771243 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 22:54:35.772162 (+ 919us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 22:54:35.772163 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 22:54:35.773003 (+ 840us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 22:54:35.773006 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 22:54:35.773450 (+ 444us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 22:54:35.773451 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 22:54:35.773511 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 22:54:35.773988 (+ 477us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 22:54:35.773993 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0929 22:54:35.775511 (+ 1518us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 22:54:35.776855 (+ 1344us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 22:54:35.776856 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 22:54:35.776856 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 22:54:35.777009 (+ 153us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 22:54:35.777169 (+ 160us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 22:54:35.777170 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 22:54:35.777170 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 22:54:35.777225 (+ 55us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 22:54:35.777468 (+ 243us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 22:54:35.777470 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 22:54:35.777515 (+ 45us) client_negotiation.cc:769] Sending connection context
0929 22:54:35.777573 (+ 58us) client_negotiation.cc:240] Negotiation successful
0929 22:54:35.777674 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":35}
I20260929 22:54:35.777776 4911 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:54:35.770769 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38208 (local address 127.14.147.252:34875)
0929 22:54:35.770825 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0929 22:54:35.770827 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 22:54:35.770839 (+ 12us) server_negotiation.cc:407] Connection header received
0929 22:54:35.770895 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 22:54:35.770896 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 22:54:35.770906 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 22:54:35.770980 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0929 22:54:35.771307 (+ 327us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:54:35.772083 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:54:35.773090 (+ 1007us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:54:35.773387 (+ 297us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:54:35.773442 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 22:54:35.775600 (+ 2158us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 22:54:35.775601 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 22:54:35.775602 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 22:54:35.775610 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0929 22:54:35.776781 (+ 1171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 22:54:35.777068 (+ 287us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 22:54:35.777069 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 22:54:35.777069 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 22:54:35.777113 (+ 44us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 22:54:35.777272 (+ 159us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 22:54:35.777273 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 22:54:35.777274 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 22:54:35.777411 (+ 137us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 22:54:35.777468 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0929 22:54:35.777599 (+ 131us) server_negotiation.cc:299] Negotiation successful
0929 22:54:35.777674 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260929 22:54:35.779668 4758 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 746ae7c9cbf64a518827cee4606d5425: Bootstrap starting.
I20260929 22:54:35.780359 4758 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 746ae7c9cbf64a518827cee4606d5425: Neither blocks nor log segments found. Creating new log.
I20260929 22:54:35.780613 4758 log.cc:824] T 00000000000000000000000000000000 P 746ae7c9cbf64a518827cee4606d5425: Log is configured to *not* fsync() on all Append() calls
I20260929 22:54:35.781291 4758 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 746ae7c9cbf64a518827cee4606d5425: No bootstrap required, opened a new log
I20260929 22:54:35.782285 4758 raft_consensus.cc:348] T 00000000000000000000000000000000 P 746ae7c9cbf64a518827cee4606d5425 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "746ae7c9cbf64a518827cee4606d5425" member_type: VOTER last_known_addr { host: "127.14.147.254" port: 44207 } } peers { permanent_uuid: "c1c8536b50cc416ab00186085ca75959" member_type: VOTER last_known_addr { host: "127.14.147.253" port: 43729 } } peers { permanent_uuid: "c5419fdb9421499aaedbe20fa12d63e2" member_type: VOTER last_known_addr { host: "127.14.147.252" port: 34875 } }
I20260929 22:54:35.782401 4758 raft_consensus.cc:729] T 00000000000000000000000000000000 P 746ae7c9cbf64a518827cee4606d5425 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 746ae7c9cbf64a518827cee4606d5425, State: Initialized, Role: FOLLOWER
I20260929 22:54:35.782476 4758 consensus_queue.cc:260] T 00000000000000000000000000000000 P 746ae7c9cbf64a518827cee4606d5425 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "746ae7c9cbf64a518827cee4606d5425" member_type: VOTER last_known_addr { host: "127.14.147.254" port: 44207 } } peers { permanent_uuid: "c1c8536b50cc416ab00186085ca75959" member_type: VOTER last_known_addr { host: "127.14.147.253" port: 43729 } } peers { permanent_uuid: "c5419fdb9421499aaedbe20fa12d63e2" member_type: VOTER last_known_addr { host: "127.14.147.252" port: 34875 } }
I20260929 22:54:35.782732 4928 sys_catalog.cc:455] T 00000000000000000000000000000000 P 746ae7c9cbf64a518827cee4606d5425 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "746ae7c9cbf64a518827cee4606d5425" member_type: VOTER last_known_addr { host: "127.14.147.254" port: 44207 } } peers { permanent_uuid: "c1c8536b50cc416ab00186085ca75959" member_type: VOTER last_known_addr { host: "127.14.147.253" port: 43729 } } peers { permanent_uuid: "c5419fdb9421499aaedbe20fa12d63e2" member_type: VOTER last_known_addr { host: "127.14.147.252" port: 34875 } } }
I20260929 22:54:35.782793 4758 sys_catalog.cc:564] T 00000000000000000000000000000000 P 746ae7c9cbf64a518827cee4606d5425 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 22:54:35.782850 4928 sys_catalog.cc:458] T 00000000000000000000000000000000 P 746ae7c9cbf64a518827cee4606d5425 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 22:54:35.782869 4732 raft_consensus.cc:482] T 00000000000000000000000000000000 P 746ae7c9cbf64a518827cee4606d5425 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260929 22:54:35.783015 4732 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 746ae7c9cbf64a518827cee4606d5425 [term 0 FOLLOWER]: Advancing to term 1
I20260929 22:54:35.783869 4732 raft_consensus.cc:504] T 00000000000000000000000000000000 P 746ae7c9cbf64a518827cee4606d5425 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "746ae7c9cbf64a518827cee4606d5425" member_type: VOTER last_known_addr { host: "127.14.147.254" port: 44207 } } peers { permanent_uuid: "c1c8536b50cc416ab00186085ca75959" member_type: VOTER last_known_addr { host: "127.14.147.253" port: 43729 } } peers { permanent_uuid: "c5419fdb9421499aaedbe20fa12d63e2" member_type: VOTER last_known_addr { host: "127.14.147.252" port: 34875 } }
I20260929 22:54:35.784791 4882 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "746ae7c9cbf64a518827cee4606d5425" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "c5419fdb9421499aaedbe20fa12d63e2"
I20260929 22:54:35.784922 4882 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c5419fdb9421499aaedbe20fa12d63e2 [term 0 FOLLOWER]: Advancing to term 1
W20260929 22:54:35.785059 4939 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 746ae7c9cbf64a518827cee4606d5425: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 22:54:35.785176 4939 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260929 22:54:35.785878 4882 raft_consensus.cc:2454] T 00000000000000000000000000000000 P c5419fdb9421499aaedbe20fa12d63e2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 746ae7c9cbf64a518827cee4606d5425 in term 1.
I20260929 22:54:35.786173 4709 leader_election.cc:302] T 00000000000000000000000000000000 P 746ae7c9cbf64a518827cee4606d5425 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 746ae7c9cbf64a518827cee4606d5425, c5419fdb9421499aaedbe20fa12d63e2; no voters:
I20260929 22:54:35.786300 4928 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 746ae7c9cbf64a518827cee4606d5425 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 22:54:35.786437 4928 raft_consensus.cc:686] T 00000000000000000000000000000000 P 746ae7c9cbf64a518827cee4606d5425 [term 1 LEADER]: Becoming Leader. State: Replica: 746ae7c9cbf64a518827cee4606d5425, State: Running, Role: LEADER
I20260929 22:54:35.786530 4928 consensus_queue.cc:237] T 00000000000000000000000000000000 P 746ae7c9cbf64a518827cee4606d5425 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "746ae7c9cbf64a518827cee4606d5425" member_type: VOTER last_known_addr { host: "127.14.147.254" port: 44207 } } peers { permanent_uuid: "c1c8536b50cc416ab00186085ca75959" member_type: VOTER last_known_addr { host: "127.14.147.253" port: 43729 } } peers { permanent_uuid: "c5419fdb9421499aaedbe20fa12d63e2" member_type: VOTER last_known_addr { host: "127.14.147.252" port: 34875 } }
I20260929 22:54:35.786943 4928 sys_catalog.cc:455] T 00000000000000000000000000000000 P 746ae7c9cbf64a518827cee4606d5425 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 746ae7c9cbf64a518827cee4606d5425. Latest consensus state: current_term: 1 leader_uuid: "746ae7c9cbf64a518827cee4606d5425" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "746ae7c9cbf64a518827cee4606d5425" member_type: VOTER last_known_addr { host: "127.14.147.254" port: 44207 } } peers { permanent_uuid: "c1c8536b50cc416ab00186085ca75959" member_type: VOTER last_known_addr { host: "127.14.147.253" port: 43729 } } peers { permanent_uuid: "c5419fdb9421499aaedbe20fa12d63e2" member_type: VOTER last_known_addr { host: "127.14.147.252" port: 34875 } } }
I20260929 22:54:35.787046 4928 sys_catalog.cc:458] T 00000000000000000000000000000000 P 746ae7c9cbf64a518827cee4606d5425 [sys.catalog]: This master's current role is: LEADER
I20260929 22:54:35.787191 4732 leader_election.cc:288] T 00000000000000000000000000000000 P 746ae7c9cbf64a518827cee4606d5425 [CANDIDATE]: Term 1 election: Requested vote from peers c1c8536b50cc416ab00186085ca75959 (127.14.147.253:43729), c5419fdb9421499aaedbe20fa12d63e2 (127.14.147.252:34875)
I20260929 22:54:35.785334 4690 master_runner.cc:186] Error getting master registration for 127.14.147.253:43729: OK, instance_id { permanent_uuid: "c1c8536b50cc416ab00186085ca75959" instance_seqno: 1790722475572012 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260929 22:54:35.787300 4690 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260929 22:54:35.791591 4943 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 22:54:35.791826 4943 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 22:54:35.796792 4911 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:54:35.788563 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38212 (local address 127.14.147.252:34875)
0929 22:54:35.788619 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0929 22:54:35.788620 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0929 22:54:35.788924 (+ 304us) server_negotiation.cc:407] Connection header received
0929 22:54:35.789011 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 22:54:35.789012 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 22:54:35.789022 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 22:54:35.789049 (+ 27us) server_negotiation.cc:226] Negotiated authn=SASL
0929 22:54:35.789626 (+ 577us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:54:35.790310 (+ 684us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:54:35.791391 (+ 1081us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:54:35.791763 (+ 372us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:54:35.791806 (+ 43us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 22:54:35.793488 (+ 1682us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 22:54:35.793489 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 22:54:35.793490 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 22:54:35.793497 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0929 22:54:35.795526 (+ 2029us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 22:54:35.795928 (+ 402us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 22:54:35.795930 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 22:54:35.795930 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 22:54:35.795984 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 22:54:35.796190 (+ 206us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 22:54:35.796191 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 22:54:35.796191 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 22:54:35.796322 (+ 131us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 22:54:35.796392 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0929 22:54:35.796614 (+ 222us) server_negotiation.cc:299] Negotiation successful
0929 22:54:35.796694 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260929 22:54:35.796933 4837 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:54:35.788464 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55832 (local address 127.14.147.253:43729)
0929 22:54:35.788505 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0929 22:54:35.788507 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 22:54:35.788751 (+ 244us) server_negotiation.cc:407] Connection header received
0929 22:54:35.788840 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 22:54:35.788841 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 22:54:35.788851 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 22:54:35.788881 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0929 22:54:35.789527 (+ 646us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:54:35.790289 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:54:35.791392 (+ 1103us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:54:35.791764 (+ 372us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:54:35.791806 (+ 42us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 22:54:35.793498 (+ 1692us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 22:54:35.793499 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 22:54:35.793499 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 22:54:35.793506 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0929 22:54:35.795310 (+ 1804us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 22:54:35.795968 (+ 658us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 22:54:35.795969 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 22:54:35.795969 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 22:54:35.796035 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 22:54:35.796288 (+ 253us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 22:54:35.796289 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 22:54:35.796289 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 22:54:35.796424 (+ 135us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 22:54:35.796486 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0929 22:54:35.796775 (+ 289us) server_negotiation.cc:299] Negotiation successful
0929 22:54:35.796850 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260929 22:54:35.797632 4762 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:54:35.788323 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47574 (local address 127.14.147.254:44207)
0929 22:54:35.788361 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0929 22:54:35.788363 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 22:54:35.788476 (+ 113us) server_negotiation.cc:407] Connection header received
0929 22:54:35.788619 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 22:54:35.788620 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 22:54:35.788629 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 22:54:35.788699 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0929 22:54:35.789920 (+ 1221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:54:35.790645 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:54:35.792117 (+ 1472us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:54:35.792389 (+ 272us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:54:35.792429 (+ 40us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 22:54:35.794679 (+ 2250us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 22:54:35.794681 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 22:54:35.794681 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 22:54:35.794689 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0929 22:54:35.796255 (+ 1566us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 22:54:35.796621 (+ 366us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 22:54:35.796622 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 22:54:35.796622 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 22:54:35.796680 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 22:54:35.796872 (+ 192us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 22:54:35.796873 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 22:54:35.796873 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 22:54:35.797014 (+ 141us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 22:54:35.797081 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0929 22:54:35.797467 (+ 386us) server_negotiation.cc:299] Negotiation successful
0929 22:54:35.797547 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260929 22:54:35.804571 4882 raft_consensus.cc:1261] T 00000000000000000000000000000000 P c5419fdb9421499aaedbe20fa12d63e2 [term 1 FOLLOWER]: Refusing update from remote peer 746ae7c9cbf64a518827cee4606d5425: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 22:54:35.804980 4941 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 746ae7c9cbf64a518827cee4606d5425 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c5419fdb9421499aaedbe20fa12d63e2" member_type: VOTER last_known_addr { host: "127.14.147.252" port: 34875 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 22:54:35.805930 4833 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.14.147.252" port: 34875 } attempt: 3
I20260929 22:54:35.806751 4914 sys_catalog.cc:455] T 00000000000000000000000000000000 P c5419fdb9421499aaedbe20fa12d63e2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 746ae7c9cbf64a518827cee4606d5425. Latest consensus state: current_term: 1 leader_uuid: "746ae7c9cbf64a518827cee4606d5425" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "746ae7c9cbf64a518827cee4606d5425" member_type: VOTER last_known_addr { host: "127.14.147.254" port: 44207 } } peers { permanent_uuid: "c1c8536b50cc416ab00186085ca75959" member_type: VOTER last_known_addr { host: "127.14.147.253" port: 43729 } } peers { permanent_uuid: "c5419fdb9421499aaedbe20fa12d63e2" member_type: VOTER last_known_addr { host: "127.14.147.252" port: 34875 } } }
I20260929 22:54:35.806850 4914 sys_catalog.cc:458] T 00000000000000000000000000000000 P c5419fdb9421499aaedbe20fa12d63e2 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 22:54:35.808702 4941 sys_catalog.cc:455] T 00000000000000000000000000000000 P 746ae7c9cbf64a518827cee4606d5425 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "746ae7c9cbf64a518827cee4606d5425" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "746ae7c9cbf64a518827cee4606d5425" member_type: VOTER last_known_addr { host: "127.14.147.254" port: 44207 } } peers { permanent_uuid: "c1c8536b50cc416ab00186085ca75959" member_type: VOTER last_known_addr { host: "127.14.147.253" port: 43729 } } peers { permanent_uuid: "c5419fdb9421499aaedbe20fa12d63e2" member_type: VOTER last_known_addr { host: "127.14.147.252" port: 34875 } } }
I20260929 22:54:35.808809 4941 sys_catalog.cc:458] T 00000000000000000000000000000000 P 746ae7c9cbf64a518827cee4606d5425 [sys.catalog]: This master's current role is: LEADER
I20260929 22:54:35.809331 4914 sys_catalog.cc:455] T 00000000000000000000000000000000 P c5419fdb9421499aaedbe20fa12d63e2 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "746ae7c9cbf64a518827cee4606d5425" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "746ae7c9cbf64a518827cee4606d5425" member_type: VOTER last_known_addr { host: "127.14.147.254" port: 44207 } } peers { permanent_uuid: "c1c8536b50cc416ab00186085ca75959" member_type: VOTER last_known_addr { host: "127.14.147.253" port: 43729 } } peers { permanent_uuid: "c5419fdb9421499aaedbe20fa12d63e2" member_type: VOTER last_known_addr { host: "127.14.147.252" port: 34875 } } }
I20260929 22:54:35.809423 4914 sys_catalog.cc:458] T 00000000000000000000000000000000 P c5419fdb9421499aaedbe20fa12d63e2 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 22:54:35.810058 4951 mvcc.cc:205] Tried to move back new op lower bound from 7334799260892459008 to 7334799260822290432. Current Snapshot: MvccSnapshot[applied={T|T < 7334799260892459008}]
I20260929 22:54:35.811398 4952 mvcc.cc:205] Tried to move back new op lower bound from 7334799260892459008 to 7334799260822290432. Current Snapshot: MvccSnapshot[applied={T|T < 7334799260892459008}]
Sep 29 22:54:35 dist-test-slave-98mk krb5kdc[4677](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790722475, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.147.253@KRBTEST.COM for kudu/127.14.147.252@KRBTEST.COM
I20260929 22:54:35.815950 4835 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:54:35.806259 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.147.252:34875 (local address 127.0.0.1:38220)
0929 22:54:35.806362 (+ 103us) negotiation.cc:107] Waiting for socket to connect
0929 22:54:35.806373 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0929 22:54:35.806412 (+ 39us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 22:54:35.806776 (+ 364us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 22:54:35.806777 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 22:54:35.806966 (+ 189us) client_negotiation.cc:189] Negotiated authn=SASL
0929 22:54:35.807018 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 22:54:35.807020 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 22:54:35.808126 (+ 1106us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 22:54:35.808128 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 22:54:35.808975 (+ 847us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 22:54:35.808979 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 22:54:35.810366 (+ 1387us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 22:54:35.810367 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 22:54:35.810449 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 22:54:35.810945 (+ 496us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 22:54:35.810951 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0929 22:54:35.812675 (+ 1724us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 22:54:35.814776 (+ 2101us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 22:54:35.814778 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 22:54:35.814778 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 22:54:35.815015 (+ 237us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 22:54:35.815236 (+ 221us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 22:54:35.815237 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 22:54:35.815237 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 22:54:35.815302 (+ 65us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 22:54:35.815594 (+ 292us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 22:54:35.815595 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 22:54:35.815656 (+ 61us) client_negotiation.cc:769] Sending connection context
0929 22:54:35.815726 (+ 70us) client_negotiation.cc:240] Negotiation successful
0929 22:54:35.815841 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":82}
I20260929 22:54:35.816605 4911 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:54:35.806483 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38220 (local address 127.14.147.252:34875)
0929 22:54:35.806523 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0929 22:54:35.806525 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 22:54:35.806539 (+ 14us) server_negotiation.cc:407] Connection header received
0929 22:54:35.806559 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 22:54:35.806559 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 22:54:35.806569 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 22:54:35.806606 (+ 37us) server_negotiation.cc:226] Negotiated authn=SASL
0929 22:54:35.807129 (+ 523us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:54:35.807923 (+ 794us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:54:35.809844 (+ 1921us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:54:35.810167 (+ 323us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:54:35.810228 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 22:54:35.812791 (+ 2563us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 22:54:35.812793 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 22:54:35.812793 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 22:54:35.812801 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0929 22:54:35.814677 (+ 1876us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 22:54:35.815102 (+ 425us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 22:54:35.815103 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 22:54:35.815104 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 22:54:35.815173 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 22:54:35.815358 (+ 185us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 22:54:35.815359 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 22:54:35.815359 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 22:54:35.815531 (+ 172us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 22:54:35.815599 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0929 22:54:35.815756 (+ 157us) server_negotiation.cc:299] Negotiation successful
0929 22:54:35.816507 (+ 751us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260929 22:54:35.818698 4833 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c1c8536b50cc416ab00186085ca75959: Bootstrap starting.
I20260929 22:54:35.819476 4833 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c1c8536b50cc416ab00186085ca75959: Neither blocks nor log segments found. Creating new log.
I20260929 22:54:35.819792 4833 log.cc:824] T 00000000000000000000000000000000 P c1c8536b50cc416ab00186085ca75959: Log is configured to *not* fsync() on all Append() calls
I20260929 22:54:35.820006 4943 catalog_manager.cc:1347] Generated new cluster ID: 3cd6acf4be0142d5a15acb8e5e561525
I20260929 22:54:35.820055 4943 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 22:54:35.820884 4833 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c1c8536b50cc416ab00186085ca75959: No bootstrap required, opened a new log
I20260929 22:54:35.823123 4833 raft_consensus.cc:348] T 00000000000000000000000000000000 P c1c8536b50cc416ab00186085ca75959 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "746ae7c9cbf64a518827cee4606d5425" member_type: VOTER last_known_addr { host: "127.14.147.254" port: 44207 } } peers { permanent_uuid: "c1c8536b50cc416ab00186085ca75959" member_type: VOTER last_known_addr { host: "127.14.147.253" port: 43729 } } peers { permanent_uuid: "c5419fdb9421499aaedbe20fa12d63e2" member_type: VOTER last_known_addr { host: "127.14.147.252" port: 34875 } }
I20260929 22:54:35.823290 4833 raft_consensus.cc:729] T 00000000000000000000000000000000 P c1c8536b50cc416ab00186085ca75959 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c1c8536b50cc416ab00186085ca75959, State: Initialized, Role: FOLLOWER
I20260929 22:54:35.823378 4833 consensus_queue.cc:260] T 00000000000000000000000000000000 P c1c8536b50cc416ab00186085ca75959 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "746ae7c9cbf64a518827cee4606d5425" member_type: VOTER last_known_addr { host: "127.14.147.254" port: 44207 } } peers { permanent_uuid: "c1c8536b50cc416ab00186085ca75959" member_type: VOTER last_known_addr { host: "127.14.147.253" port: 43729 } } peers { permanent_uuid: "c5419fdb9421499aaedbe20fa12d63e2" member_type: VOTER last_known_addr { host: "127.14.147.252" port: 34875 } }
I20260929 22:54:35.823815 4833 sys_catalog.cc:564] T 00000000000000000000000000000000 P c1c8536b50cc416ab00186085ca75959 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 22:54:35.824199 4806 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c1c8536b50cc416ab00186085ca75959 [term 0 FOLLOWER]: Advancing to term 1
I20260929 22:54:35.825145 4806 raft_consensus.cc:1261] T 00000000000000000000000000000000 P c1c8536b50cc416ab00186085ca75959 [term 1 FOLLOWER]: Refusing update from remote peer 746ae7c9cbf64a518827cee4606d5425: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 22:54:35.825538 4807 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "746ae7c9cbf64a518827cee4606d5425" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "c1c8536b50cc416ab00186085ca75959"
I20260929 22:54:35.825852 4928 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 746ae7c9cbf64a518827cee4606d5425 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c1c8536b50cc416ab00186085ca75959" member_type: VOTER last_known_addr { host: "127.14.147.253" port: 43729 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 22:54:35.826339 4807 raft_consensus.cc:2454] T 00000000000000000000000000000000 P c1c8536b50cc416ab00186085ca75959 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 746ae7c9cbf64a518827cee4606d5425 in term 1.
I20260929 22:54:35.828212 4943 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 22:54:35.828943 4943 catalog_manager.cc:1504] Loading token signing keys...
W20260929 22:54:35.832216 4972 catalog_manager.cc:1558] T 00000000000000000000000000000000 P c1c8536b50cc416ab00186085ca75959: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 22:54:35.832294 4972 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260929 22:54:35.832440 4957 sys_catalog.cc:455] T 00000000000000000000000000000000 P c1c8536b50cc416ab00186085ca75959 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "746ae7c9cbf64a518827cee4606d5425" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "746ae7c9cbf64a518827cee4606d5425" member_type: VOTER last_known_addr { host: "127.14.147.254" port: 44207 } } peers { permanent_uuid: "c1c8536b50cc416ab00186085ca75959" member_type: VOTER last_known_addr { host: "127.14.147.253" port: 43729 } } peers { permanent_uuid: "c5419fdb9421499aaedbe20fa12d63e2" member_type: VOTER last_known_addr { host: "127.14.147.252" port: 34875 } } }
I20260929 22:54:35.832616 4958 sys_catalog.cc:455] T 00000000000000000000000000000000 P c1c8536b50cc416ab00186085ca75959 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 746ae7c9cbf64a518827cee4606d5425. Latest consensus state: current_term: 1 leader_uuid: "746ae7c9cbf64a518827cee4606d5425" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "746ae7c9cbf64a518827cee4606d5425" member_type: VOTER last_known_addr { host: "127.14.147.254" port: 44207 } } peers { permanent_uuid: "c1c8536b50cc416ab00186085ca75959" member_type: VOTER last_known_addr { host: "127.14.147.253" port: 43729 } } peers { permanent_uuid: "c5419fdb9421499aaedbe20fa12d63e2" member_type: VOTER last_known_addr { host: "127.14.147.252" port: 34875 } } }
I20260929 22:54:35.832679 4958 sys_catalog.cc:458] T 00000000000000000000000000000000 P c1c8536b50cc416ab00186085ca75959 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 22:54:35.832731 4958 sys_catalog.cc:455] T 00000000000000000000000000000000 P c1c8536b50cc416ab00186085ca75959 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "746ae7c9cbf64a518827cee4606d5425" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "746ae7c9cbf64a518827cee4606d5425" member_type: VOTER last_known_addr { host: "127.14.147.254" port: 44207 } } peers { permanent_uuid: "c1c8536b50cc416ab00186085ca75959" member_type: VOTER last_known_addr { host: "127.14.147.253" port: 43729 } } peers { permanent_uuid: "c5419fdb9421499aaedbe20fa12d63e2" member_type: VOTER last_known_addr { host: "127.14.147.252" port: 34875 } } }
I20260929 22:54:35.832777 4958 sys_catalog.cc:458] T 00000000000000000000000000000000 P c1c8536b50cc416ab00186085ca75959 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 22:54:35.833261 4959 mvcc.cc:205] Tried to move back new op lower bound from 7334799260892459008 to 7334799260822290432. Current Snapshot: MvccSnapshot[applied={T|T < 7334799260892459008}]
I20260929 22:54:35.832517 4957 sys_catalog.cc:458] T 00000000000000000000000000000000 P c1c8536b50cc416ab00186085ca75959 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 22:54:35.835290 4941 sys_catalog.cc:455] T 00000000000000000000000000000000 P 746ae7c9cbf64a518827cee4606d5425 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "746ae7c9cbf64a518827cee4606d5425" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "746ae7c9cbf64a518827cee4606d5425" member_type: VOTER last_known_addr { host: "127.14.147.254" port: 44207 } } peers { permanent_uuid: "c1c8536b50cc416ab00186085ca75959" member_type: VOTER last_known_addr { host: "127.14.147.253" port: 43729 } } peers { permanent_uuid: "c5419fdb9421499aaedbe20fa12d63e2" member_type: VOTER last_known_addr { host: "127.14.147.252" port: 34875 } } }
I20260929 22:54:35.835405 4941 sys_catalog.cc:458] T 00000000000000000000000000000000 P 746ae7c9cbf64a518827cee4606d5425 [sys.catalog]: This master's current role is: LEADER
I20260929 22:54:35.844776 4943 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 746ae7c9cbf64a518827cee4606d5425: Generated new TSK 0
I20260929 22:54:35.845010 4943 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260929 22:54:35.870879 4762 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 22:54:35.869786 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47576 (local address 127.14.147.254:44207)
0929 22:54:35.869835 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0929 22:54:35.869838 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 22:54:35.870142 (+ 304us) server_negotiation.cc:407] Connection header received
0929 22:54:35.870245 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 22:54:35.870246 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 22:54:35.870257 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 22:54:35.870288 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0929 22:54:35.870790 (+ 502us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47576: BlockingRecv error: recv got EOF from 127.0.0.1:47576 (error 108)
Metrics: {"server-negotiator.queue_time_us":23}
W20260929 22:54:35.872002 4911 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 22:54:35.871320 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38222 (local address 127.14.147.252:34875)
0929 22:54:35.871368 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0929 22:54:35.871370 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 22:54:35.871379 (+ 9us) server_negotiation.cc:407] Connection header received
0929 22:54:35.871407 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 22:54:35.871408 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 22:54:35.871468 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 22:54:35.871542 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0929 22:54:35.871937 (+ 395us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38222: BlockingRecv error: recv got EOF from 127.0.0.1:38222 (error 108)
Metrics: {"server-negotiator.queue_time_us":24}
W20260929 22:54:35.872413 4837 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 22:54:35.871582 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55836 (local address 127.14.147.253:43729)
0929 22:54:35.871623 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0929 22:54:35.871625 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 22:54:35.871845 (+ 220us) server_negotiation.cc:407] Connection header received
0929 22:54:35.871970 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 22:54:35.871971 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 22:54:35.871982 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 22:54:35.872010 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0929 22:54:35.872350 (+ 340us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:55836: BlockingRecv error: recv got EOF from 127.0.0.1:55836 (error 108)
Metrics: {"server-negotiator.queue_time_us":21}
W20260929 22:54:35.874680 4837 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 22:54:35.873964 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55846 (local address 127.14.147.253:43729)
0929 22:54:35.873999 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0929 22:54:35.874001 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 22:54:35.874164 (+ 163us) server_negotiation.cc:407] Connection header received
0929 22:54:35.874268 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 22:54:35.874269 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 22:54:35.874288 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 22:54:35.874316 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0929 22:54:35.874630 (+ 314us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:55846: BlockingRecv error: recv got EOF from 127.0.0.1:55846 (error 108)
Metrics: {"server-negotiator.queue_time_us":16}
W20260929 22:54:35.877678 4837 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 22:54:35.876927 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55854 (local address 127.14.147.253:43729)
0929 22:54:35.876961 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0929 22:54:35.876963 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 22:54:35.877168 (+ 205us) server_negotiation.cc:407] Connection header received
0929 22:54:35.877261 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 22:54:35.877262 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 22:54:35.877272 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 22:54:35.877300 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0929 22:54:35.877627 (+ 327us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:55854: BlockingRecv error: recv got EOF from 127.0.0.1:55854 (error 108)
Metrics: {"server-negotiator.queue_time_us":16}
W20260929 22:54:35.881650 4837 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 22:54:35.880858 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55868 (local address 127.14.147.253:43729)
0929 22:54:35.880902 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0929 22:54:35.880904 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 22:54:35.881112 (+ 208us) server_negotiation.cc:407] Connection header received
0929 22:54:35.881201 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 22:54:35.881202 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 22:54:35.881214 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 22:54:35.881245 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0929 22:54:35.881596 (+ 351us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:55868: BlockingRecv error: recv got EOF from 127.0.0.1:55868 (error 108)
Metrics: {"server-negotiator.queue_time_us":17}
W20260929 22:54:35.888072 4837 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 22:54:35.887095 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55870 (local address 127.14.147.253:43729)
0929 22:54:35.887139 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0929 22:54:35.887141 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 22:54:35.887403 (+ 262us) server_negotiation.cc:407] Connection header received
0929 22:54:35.887569 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 22:54:35.887570 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 22:54:35.887584 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 22:54:35.887614 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0929 22:54:35.888011 (+ 397us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:55870: BlockingRecv error: recv got EOF from 127.0.0.1:55870 (error 108)
Metrics: {"server-negotiator.queue_time_us":21}
W20260929 22:54:35.898414 4837 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 22:54:35.897474 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55880 (local address 127.14.147.253:43729)
0929 22:54:35.897546 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0929 22:54:35.897549 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 22:54:35.897806 (+ 257us) server_negotiation.cc:407] Connection header received
0929 22:54:35.897912 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 22:54:35.897914 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 22:54:35.897926 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 22:54:35.897956 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0929 22:54:35.898349 (+ 393us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:55880: BlockingRecv error: recv got EOF from 127.0.0.1:55880 (error 108)
Metrics: {"server-negotiator.queue_time_us":22}
W20260929 22:54:35.916903 4837 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 22:54:35.915914 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55892 (local address 127.14.147.253:43729)
0929 22:54:35.915962 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0929 22:54:35.915964 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 22:54:35.916241 (+ 277us) server_negotiation.cc:407] Connection header received
0929 22:54:35.916350 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 22:54:35.916351 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 22:54:35.916364 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 22:54:35.916395 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0929 22:54:35.916840 (+ 445us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:55892: BlockingRecv error: recv got EOF from 127.0.0.1:55892 (error 108)
Metrics: {"server-negotiator.queue_time_us":22}
W20260929 22:54:35.951766 4837 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 22:54:35.950784 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55908 (local address 127.14.147.253:43729)
0929 22:54:35.950834 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0929 22:54:35.950837 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 22:54:35.951097 (+ 260us) server_negotiation.cc:407] Connection header received
0929 22:54:35.951195 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 22:54:35.951197 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 22:54:35.951209 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 22:54:35.951239 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0929 22:54:35.951700 (+ 461us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:55908: BlockingRecv error: recv got EOF from 127.0.0.1:55908 (error 108)
Metrics: {"server-negotiator.queue_time_us":20}
W20260929 22:54:36.017795 4837 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 22:54:36.017226 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55920 (local address 127.14.147.253:43729)
0929 22:54:36.017312 (+ 86us) server_negotiation.cc:206] Beginning negotiation
0929 22:54:36.017314 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 22:54:36.017357 (+ 43us) server_negotiation.cc:407] Connection header received
0929 22:54:36.017386 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 22:54:36.017387 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 22:54:36.017398 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 22:54:36.017441 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0929 22:54:36.017741 (+ 300us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:55920: BlockingRecv error: recv got EOF from 127.0.0.1:55920 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
W20260929 22:54:36.147773 4837 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 22:54:36.147114 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55928 (local address 127.14.147.253:43729)
0929 22:54:36.147233 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0929 22:54:36.147235 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 22:54:36.147245 (+ 10us) server_negotiation.cc:407] Connection header received
0929 22:54:36.147269 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 22:54:36.147270 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 22:54:36.147281 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 22:54:36.147332 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0929 22:54:36.147722 (+ 390us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:55928: BlockingRecv error: recv got EOF from 127.0.0.1:55928 (error 108)
Metrics: {"server-negotiator.queue_time_us":78}
W20260929 22:54:36.405772 4837 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 22:54:36.405185 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55942 (local address 127.14.147.253:43729)
0929 22:54:36.405243 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0929 22:54:36.405244 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0929 22:54:36.405253 (+ 9us) server_negotiation.cc:407] Connection header received
0929 22:54:36.405279 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 22:54:36.405280 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 22:54:36.405290 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 22:54:36.405340 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0929 22:54:36.405709 (+ 369us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:55942: BlockingRecv error: recv got EOF from 127.0.0.1:55942 (error 108)
Metrics: {"server-negotiator.queue_time_us":34}
I20260929 22:54:36.761560 4925 catalog_manager.cc:1259] Loaded cluster ID: 3cd6acf4be0142d5a15acb8e5e561525
I20260929 22:54:36.761650 4925 catalog_manager.cc:1552] T 00000000000000000000000000000000 P c5419fdb9421499aaedbe20fa12d63e2: loading cluster ID for follower catalog manager: success
I20260929 22:54:36.762449 4925 catalog_manager.cc:1574] T 00000000000000000000000000000000 P c5419fdb9421499aaedbe20fa12d63e2: acquiring CA information for follower catalog manager: success
I20260929 22:54:36.762748 4925 catalog_manager.cc:1602] T 00000000000000000000000000000000 P c5419fdb9421499aaedbe20fa12d63e2: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260929 22:54:36.832937 4972 catalog_manager.cc:1259] Loaded cluster ID: 3cd6acf4be0142d5a15acb8e5e561525
I20260929 22:54:36.833032 4972 catalog_manager.cc:1552] T 00000000000000000000000000000000 P c1c8536b50cc416ab00186085ca75959: loading cluster ID for follower catalog manager: success
I20260929 22:54:36.833977 4972 catalog_manager.cc:1574] T 00000000000000000000000000000000 P c1c8536b50cc416ab00186085ca75959: acquiring CA information for follower catalog manager: success
I20260929 22:54:36.834286 4972 catalog_manager.cc:1602] T 00000000000000000000000000000000 P c1c8536b50cc416ab00186085ca75959: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260929 22:54:36.921808 5039 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:54:36.919158 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55946 (local address 127.14.147.253:43729)
0929 22:54:36.919348 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0929 22:54:36.919350 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 22:54:36.919359 (+ 9us) server_negotiation.cc:407] Connection header received
0929 22:54:36.919385 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 22:54:36.919385 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 22:54:36.919493 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 22:54:36.919568 (+ 75us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 22:54:36.919746 (+ 178us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:54:36.920263 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:54:36.921025 (+ 762us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:54:36.921231 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:54:36.921287 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 22:54:36.921440 (+ 153us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 22:54:36.921553 (+ 113us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 22:54:36.921619 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0929 22:54:36.921688 (+ 69us) server_negotiation.cc:299] Negotiation successful
0929 22:54:36.921713 (+ 25us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":163,"thread_start_us":85,"threads_started":1}
I20260929 22:54:36.926160 5045 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:54:36.923596 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38226 (local address 127.14.147.252:34875)
0929 22:54:36.923690 (+ 94us) server_negotiation.cc:206] Beginning negotiation
0929 22:54:36.923692 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 22:54:36.923701 (+ 9us) server_negotiation.cc:407] Connection header received
0929 22:54:36.923779 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 22:54:36.923780 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 22:54:36.923815 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 22:54:36.923874 (+ 59us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 22:54:36.923946 (+ 72us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:54:36.924460 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:54:36.925530 (+ 1070us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:54:36.925732 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:54:36.925779 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 22:54:36.925893 (+ 114us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 22:54:36.925995 (+ 102us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 22:54:36.926045 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0929 22:54:36.926070 (+ 25us) server_negotiation.cc:299] Negotiation successful
0929 22:54:36.926094 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75,"thread_start_us":48,"threads_started":1}
I20260929 22:54:36.930516 5045 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:54:36.927861 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38230 (local address 127.14.147.252:34875)
0929 22:54:36.927911 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0929 22:54:36.927914 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 22:54:36.928191 (+ 277us) server_negotiation.cc:407] Connection header received
0929 22:54:36.928218 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 22:54:36.928219 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 22:54:36.928230 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 22:54:36.928274 (+ 44us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 22:54:36.928438 (+ 164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:54:36.929148 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:54:36.929843 (+ 695us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:54:36.930047 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:54:36.930099 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 22:54:36.930272 (+ 173us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 22:54:36.930306 (+ 34us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 22:54:36.930361 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0929 22:54:36.930413 (+ 52us) server_negotiation.cc:299] Negotiation successful
0929 22:54:36.930437 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260929 22:54:36.931000 5039 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:54:36.927799 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55950 (local address 127.14.147.253:43729)
0929 22:54:36.927928 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0929 22:54:36.927930 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 22:54:36.928038 (+ 108us) server_negotiation.cc:407] Connection header received
0929 22:54:36.928118 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 22:54:36.928119 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 22:54:36.928129 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 22:54:36.928165 (+ 36us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 22:54:36.928278 (+ 113us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:54:36.929000 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:54:36.930085 (+ 1085us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:54:36.930366 (+ 281us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:54:36.930423 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 22:54:36.930761 (+ 338us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 22:54:36.930797 (+ 36us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 22:54:36.930821 (+ 24us) server_negotiation.cc:1035] Waiting for connection context
0929 22:54:36.930900 (+ 79us) server_negotiation.cc:299] Negotiation successful
0929 22:54:36.930928 (+ 28us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":105}
I20260929 22:54:36.931231 5051 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:54:36.927828 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47590 (local address 127.14.147.254:44207)
0929 22:54:36.928001 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0929 22:54:36.928002 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0929 22:54:36.928055 (+ 53us) server_negotiation.cc:407] Connection header received
0929 22:54:36.928242 (+ 187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 22:54:36.928243 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 22:54:36.928293 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 22:54:36.928367 (+ 74us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 22:54:36.928499 (+ 132us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:54:36.929059 (+ 560us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:54:36.930087 (+ 1028us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:54:36.930320 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:54:36.930371 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 22:54:36.930581 (+ 210us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 22:54:36.930814 (+ 233us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 22:54:36.930943 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0929 22:54:36.931049 (+ 106us) server_negotiation.cc:299] Negotiation successful
0929 22:54:36.931159 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":154,"thread_start_us":56,"threads_started":1}
I20260929 22:54:36.936329 14927 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhWOCLk/build/release/bin/kudu with pid 4690
I20260929 22:54:36.940905 14927 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhWOCLk/build/release/bin/kudu with pid 4765
I20260929 22:54:36.945050 14927 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhWOCLk/build/release/bin/kudu with pid 4840
2026-09-29T22:54:36Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0 (2728 ms)
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1
Loading random data
Initializing database '/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790722213368738-14927-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 22:54:36 dist-test-slave-98mk krb5kdc[5056](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 22:54:36 dist-test-slave-98mk krb5kdc[5056](info): set up 2 sockets
Sep 29 22:54:36 dist-test-slave-98mk krb5kdc[5056](info): commencing operation
krb5kdc: starting...
W20260929 22:54:37.892154 14927 mini_kdc.cc:121] Time spent starting KDC: real 0.928s user 0.005s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790722213368738-14927-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790722213368738-14927-0/krb5kdc/test-user.keytab.
Sep 29 22:54:37 dist-test-slave-98mk krb5kdc[5056](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790722477, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T22:54:37Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T22:54:37Z Disabled control of system clock
WARNING: no policy specified for kudu/127.14.147.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.147.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.147.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790722213368738-14927-0/krb5kdc/kudu_127.14.147.254.keytab.
Entry for principal kudu/127.14.147.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790722213368738-14927-0/krb5kdc/kudu_127.14.147.254.keytab.
I20260929 22:54:38.005964 14927 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhWOCLk/build/release/bin/kudu
/tmp/dist-test-taskhWOCLk/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790722213368738-14927-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790722213368738-14927-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790722213368738-14927-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790722213368738-14927-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.147.254:35079
--webserver_interface=127.14.147.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790722213368738-14927-0/krb5kdc/kudu_127.14.147.254.keytab
--principal=kudu/127.14.147.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.147.212:43383
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.147.254:35079,127.14.147.253:37273,127.14.147.252:45813
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790722213368738-14927-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790722213368738-14927-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790722213368738-14927-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 22:54:38.086463 5069 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260929 22:54:38.086735 5069 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 22:54:38.086762 5069 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 22:54:38.086789 5069 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 22:54:38.088050 5069 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 22:54:38.088100 5069 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 22:54:38.088119 5069 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 22:54:38.088135 5069 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 22:54:38.088150 5069 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 22:54:38.089762 5069 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.147.212:43383
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790722213368738-14927-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790722213368738-14927-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.14.147.254:35079,127.14.147.253:37273,127.14.147.252:45813
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790722213368738-14927-0/krb5kdc/kudu_127.14.147.254.keytab
--principal=kudu/127.14.147.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.147.254:35079
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790722213368738-14927-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.14.147.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5069
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790722213368738-14927-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 21:58:54 UTC on ccycloud.quasar-xguoti.root.comops.site
build id 4681390
I20260929 22:54:38.090018 5069 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 22:54:38.090242 5069 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 22:54:38.094826 5074 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 22:54:38.094873 5077 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 22:54:38.094965 5069 server_base.cc:1034] running on GCE node
W20260929 22:54:38.094869 5075 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 22:54:38.095271 5069 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 22:54:38.095618 5069 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 22:54:38.096788 5069 hybrid_clock.cc:648] HybridClock initialized: now 1790722478096769 us; error 35 us; skew 500 ppm
Sep 29 22:54:38 dist-test-slave-98mk krb5kdc[5056](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790722478, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.147.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 22:54:38.099282 5069 init.cc:377] Logged in from keytab as kudu/127.14.147.254@KRBTEST.COM (short username kudu)
I20260929 22:54:38.099761 5069 webserver.cc:460] Webserver started at http://127.14.147.254:40695/ using document root <none> and password file <none>
I20260929 22:54:38.099965 5069 fs_manager.cc:359] Metadata directory not provided
I20260929 22:54:38.100004 5069 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 22:54:38.100096 5069 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 22:54:38.101200 5069 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790722213368738-14927-0/minicluster-data/master-0/data/instance:
uuid: "d61c2709f5924100bdfd4070c8cc7d75"
format_stamp: "Formatted at 2026-09-29 22:54:38 on dist-test-slave-98mk"
server_key: "024dcd652feaf45fecd135a9f995a4fa"
server_key_iv: "da4248af10cc75d71d5025ffee572e3c"
server_key_version: "encryptionkey@0"
I20260929 22:54:38.101495 5069 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790722213368738-14927-0/minicluster-data/master-0/wal/instance:
uuid: "d61c2709f5924100bdfd4070c8cc7d75"
format_stamp: "Formatted at 2026-09-29 22:54:38 on dist-test-slave-98mk"
server_key: "024dcd652feaf45fecd135a9f995a4fa"
server_key_iv: "da4248af10cc75d71d5025ffee572e3c"
server_key_version: "encryptionkey@0"
I20260929 22:54:38.103012 5069 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260929 22:54:38.103760 5084 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 22:54:38.103925 5069 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260929 22:54:38.104012 5069 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790722213368738-14927-0/minicluster-data/master-0/data,/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790722213368738-14927-0/minicluster-data/master-0/wal
uuid: "d61c2709f5924100bdfd4070c8cc7d75"
format_stamp: "Formatted at 2026-09-29 22:54:38 on dist-test-slave-98mk"
server_key: "024dcd652feaf45fecd135a9f995a4fa"
server_key_iv: "da4248af10cc75d71d5025ffee572e3c"
server_key_version: "encryptionkey@0"
I20260929 22:54:38.104079 5069 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790722213368738-14927-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790722213368738-14927-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790722213368738-14927-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 22:54:38.128428 5069 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 22:54:38.130554 5069 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 22:54:38.130671 5069 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 22:54:38.134402 5069 rpc_server.cc:307] RPC server started. Bound to: 127.14.147.254:35079
I20260929 22:54:38.134469 5136 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.147.254:35079 every 8 connection(s)
I20260929 22:54:38.134783 5069 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790722213368738-14927-0/minicluster-data/master-0/data/info.pb
I20260929 22:54:38.135762 5137 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 22:54:38.137315 5137 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.14.147.254" port: 35079 } has no permanent_uuid. Determining permanent_uuid...
I20260929 22:54:38.141031 14927 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhWOCLk/build/release/bin/kudu as pid 5069
I20260929 22:54:38.141127 14927 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790722213368738-14927-0/minicluster-data/master-0/wal/instance
I20260929 22:54:38.141310 14927 external_mini_cluster.cc:1428] Setting key 2867e74f05c0de75c6fb1f83d3bf8ed0
Sep 29 22:54:38 dist-test-slave-98mk krb5kdc[5056](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790722478, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.147.254@KRBTEST.COM for kudu/127.14.147.254@KRBTEST.COM
I20260929 22:54:38.147084 5140 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:54:38.138164 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38286 (local address 127.14.147.254:35079)
0929 22:54:38.138273 (+ 109us) server_negotiation.cc:206] Beginning negotiation
0929 22:54:38.138276 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 22:54:38.138303 (+ 27us) server_negotiation.cc:407] Connection header received
0929 22:54:38.138832 (+ 529us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 22:54:38.138834 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 22:54:38.138850 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 22:54:38.138923 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0929 22:54:38.139510 (+ 587us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:54:38.140349 (+ 839us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:54:38.141260 (+ 911us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:54:38.141592 (+ 332us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:54:38.141691 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 22:54:38.144095 (+ 2404us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 22:54:38.144106 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 22:54:38.144106 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 22:54:38.144114 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0929 22:54:38.145514 (+ 1400us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 22:54:38.145979 (+ 465us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 22:54:38.145981 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 22:54:38.145982 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 22:54:38.146069 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 22:54:38.146295 (+ 226us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 22:54:38.146295 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 22:54:38.146296 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 22:54:38.146510 (+ 214us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 22:54:38.146572 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0929 22:54:38.146807 (+ 235us) server_negotiation.cc:299] Negotiation successful
0929 22:54:38.146905 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82,"thread_start_us":42,"threads_started":1}
I20260929 22:54:38.147114 5139 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:54:38.137993 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.147.254:35079 (local address 127.0.0.1:38286)
0929 22:54:38.138199 (+ 206us) negotiation.cc:107] Waiting for socket to connect
0929 22:54:38.138239 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0929 22:54:38.138712 (+ 473us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 22:54:38.138947 (+ 235us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 22:54:38.138949 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 22:54:38.139204 (+ 255us) client_negotiation.cc:189] Negotiated authn=SASL
0929 22:54:38.139365 (+ 161us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 22:54:38.139368 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 22:54:38.140456 (+ 1088us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 22:54:38.140457 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 22:54:38.141177 (+ 720us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 22:54:38.141179 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 22:54:38.141667 (+ 488us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 22:54:38.141668 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 22:54:38.141735 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 22:54:38.142247 (+ 512us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 22:54:38.142256 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0929 22:54:38.143975 (+ 1719us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 22:54:38.145617 (+ 1642us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 22:54:38.145626 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 22:54:38.145629 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 22:54:38.145901 (+ 272us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 22:54:38.146131 (+ 230us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 22:54:38.146132 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 22:54:38.146133 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 22:54:38.146238 (+ 105us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 22:54:38.146572 (+ 334us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 22:54:38.146579 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 22:54:38.146672 (+ 93us) client_negotiation.cc:769] Sending connection context
0929 22:54:38.146774 (+ 102us) client_negotiation.cc:240] Negotiation successful
0929 22:54:38.147000 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":157,"mutex_wait_us":56,"thread_start_us":79,"threads_started":1}
I20260929 22:54:38.147987 5137 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.14.147.253" port: 37273 } has no permanent_uuid. Determining permanent_uuid...
I20260929 22:54:38.148348 5139 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:54:38.148218 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.147.253:37273 (local address 127.0.0.1:54890)
0929 22:54:38.148266 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0929 22:54:38.148319 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.147.253:37273: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":28}
W20260929 22:54:38.148617 5088 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.14.147.253:37273: connect: Connection refused (error 111)
I20260929 22:54:38.149205 5139 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:54:38.149080 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.147.253:37273 (local address 127.0.0.1:54902)
0929 22:54:38.149141 (+ 61us) negotiation.cc:107] Waiting for socket to connect
0929 22:54:38.149179 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.147.253:37273: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260929 22:54:38.149382 5137 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.14.147.253:37273: Network error: Client connection negotiation failed: client connection to 127.14.147.253:37273: connect: Connection refused (error 111)
WARNING: no policy specified for kudu/127.14.147.253@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.147.253@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.147.253 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790722213368738-14927-0/krb5kdc/kudu_127.14.147.253.keytab.
Entry for principal kudu/127.14.147.253 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790722213368738-14927-0/krb5kdc/kudu_127.14.147.253.keytab.
I20260929 22:54:38.165742 14927 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhWOCLk/build/release/bin/kudu
/tmp/dist-test-taskhWOCLk/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790722213368738-14927-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790722213368738-14927-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790722213368738-14927-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790722213368738-14927-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.147.253:37273
--webserver_interface=127.14.147.253
--webserver_port=0
--keytab_file=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790722213368738-14927-0/krb5kdc/kudu_127.14.147.253.keytab
--principal=kudu/127.14.147.253
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.147.212:43383
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.147.254:35079,127.14.147.253:37273,127.14.147.252:45813
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790722213368738-14927-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790722213368738-14927-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790722213368738-14927-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 22:54:38.198645 5137 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.14.147.253" port: 37273 } attempt: 1
I20260929 22:54:38.199203 5139 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:54:38.199037 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.147.253:37273 (local address 127.0.0.1:54908)
0929 22:54:38.199114 (+ 77us) negotiation.cc:107] Waiting for socket to connect
0929 22:54:38.199165 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.147.253:37273: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
I20260929 22:54:38.199621 5139 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:54:38.199531 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.147.253:37273 (local address 127.0.0.1:54918)
0929 22:54:38.199572 (+ 41us) negotiation.cc:107] Waiting for socket to connect
0929 22:54:38.199601 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.147.253:37273: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":26}
W20260929 22:54:38.199765 5137 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.14.147.253:37273: Network error: Client connection negotiation failed: client connection to 127.14.147.253:37273: connect: Connection refused (error 111)
W20260929 22:54:38.248692 5144 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260929 22:54:38.248960 5144 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 22:54:38.248983 5144 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 22:54:38.249013 5144 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 22:54:38.250947 5144 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 22:54:38.251013 5144 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 22:54:38.251029 5144 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 22:54:38.251040 5144 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 22:54:38.251051 5144 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 22:54:38.253026 5144 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.147.212:43383
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790722213368738-14927-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790722213368738-14927-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.14.147.254:35079,127.14.147.253:37273,127.14.147.252:45813
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790722213368738-14927-0/krb5kdc/kudu_127.14.147.253.keytab
--principal=kudu/127.14.147.253
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.147.253:37273
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790722213368738-14927-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.14.147.253
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5144
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790722213368738-14927-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 21:58:54 UTC on ccycloud.quasar-xguoti.root.comops.site
build id 4681390
I20260929 22:54:38.253298 5144 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 22:54:38.253559 5144 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 22:54:38.259016 5149 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 22:54:38.259050 5153 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 22:54:38.259075 5150 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 22:54:38.259267 5144 server_base.cc:1034] running on GCE node
I20260929 22:54:38.259549 5144 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 22:54:38.259862 5144 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 22:54:38.261029 5144 hybrid_clock.cc:648] HybridClock initialized: now 1790722478261011 us; error 27 us; skew 500 ppm
Sep 29 22:54:38 dist-test-slave-98mk krb5kdc[5056](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790722478, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.147.253@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 22:54:38.263707 5144 init.cc:377] Logged in from keytab as kudu/127.14.147.253@KRBTEST.COM (short username kudu)
I20260929 22:54:38.264223 5144 webserver.cc:460] Webserver started at http://127.14.147.253:37755/ using document root <none> and password file <none>
I20260929 22:54:38.264458 5144 fs_manager.cc:359] Metadata directory not provided
I20260929 22:54:38.264508 5144 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 22:54:38.264633 5144 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 22:54:38.265756 5144 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790722213368738-14927-0/minicluster-data/master-1/data/instance:
uuid: "30054c4a503947e1b79aaa615e7ca7a1"
format_stamp: "Formatted at 2026-09-29 22:54:38 on dist-test-slave-98mk"
server_key: "9e61af37df60bdc9c1f38e542f18de5b"
server_key_iv: "05b1ae0b834661b0dab142a147a65d62"
server_key_version: "encryptionkey@0"
I20260929 22:54:38.266119 5144 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790722213368738-14927-0/minicluster-data/master-1/wal/instance:
uuid: "30054c4a503947e1b79aaa615e7ca7a1"
format_stamp: "Formatted at 2026-09-29 22:54:38 on dist-test-slave-98mk"
server_key: "9e61af37df60bdc9c1f38e542f18de5b"
server_key_iv: "05b1ae0b834661b0dab142a147a65d62"
server_key_version: "encryptionkey@0"
I20260929 22:54:38.267488 5144 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260929 22:54:38.268044 5137 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.14.147.253" port: 37273 } attempt: 2
I20260929 22:54:38.268639 5139 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:54:38.268427 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.147.253:37273 (local address 127.0.0.1:54922)
0929 22:54:38.268532 (+ 105us) negotiation.cc:107] Waiting for socket to connect
0929 22:54:38.268600 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.147.253:37273: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":66}
I20260929 22:54:38.268725 5160 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 22:54:38.268904 5144 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260929 22:54:38.268994 5144 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790722213368738-14927-0/minicluster-data/master-1/data,/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790722213368738-14927-0/minicluster-data/master-1/wal
uuid: "30054c4a503947e1b79aaa615e7ca7a1"
format_stamp: "Formatted at 2026-09-29 22:54:38 on dist-test-slave-98mk"
server_key: "9e61af37df60bdc9c1f38e542f18de5b"
server_key_iv: "05b1ae0b834661b0dab142a147a65d62"
server_key_version: "encryptionkey@0"
I20260929 22:54:38.269104 5139 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:54:38.268984 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.147.253:37273 (local address 127.0.0.1:54934)
0929 22:54:38.269049 (+ 65us) negotiation.cc:107] Waiting for socket to connect
0929 22:54:38.269074 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.147.253:37273: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
I20260929 22:54:38.269088 5144 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790722213368738-14927-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790722213368738-14927-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790722213368738-14927-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
W20260929 22:54:38.269197 5137 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.14.147.253:37273: Network error: Client connection negotiation failed: client connection to 127.14.147.253:37273: connect: Connection refused (error 111)
I20260929 22:54:38.293798 5144 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 22:54:38.296386 5144 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 22:54:38.296511 5144 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 22:54:38.300952 5144 rpc_server.cc:307] RPC server started. Bound to: 127.14.147.253:37273
I20260929 22:54:38.301004 5212 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.147.253:37273 every 8 connection(s)
I20260929 22:54:38.301482 5144 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790722213368738-14927-0/minicluster-data/master-1/data/info.pb
I20260929 22:54:38.302479 5213 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 22:54:38.304170 5213 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.14.147.254" port: 35079 } has no permanent_uuid. Determining permanent_uuid...
Sep 29 22:54:38 dist-test-slave-98mk krb5kdc[5056](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790722478, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.147.253@KRBTEST.COM for kudu/127.14.147.254@KRBTEST.COM
I20260929 22:54:38.311204 14927 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhWOCLk/build/release/bin/kudu as pid 5144
I20260929 22:54:38.311311 14927 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790722213368738-14927-0/minicluster-data/master-1/wal/instance
I20260929 22:54:38.311545 14927 external_mini_cluster.cc:1428] Setting key b44b851df54a97e3ebd9a47e0532f471
I20260929 22:54:38.313701 5140 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:54:38.304935 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38294 (local address 127.14.147.254:35079)
0929 22:54:38.304988 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0929 22:54:38.304990 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 22:54:38.305187 (+ 197us) server_negotiation.cc:407] Connection header received
0929 22:54:38.305707 (+ 520us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 22:54:38.305708 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 22:54:38.305723 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 22:54:38.305775 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0929 22:54:38.306451 (+ 676us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:54:38.306971 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:54:38.308000 (+ 1029us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:54:38.308217 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:54:38.308275 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 22:54:38.310323 (+ 2048us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 22:54:38.310325 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 22:54:38.310325 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 22:54:38.310334 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0929 22:54:38.312006 (+ 1672us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 22:54:38.312516 (+ 510us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 22:54:38.312517 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 22:54:38.312518 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 22:54:38.312592 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 22:54:38.312975 (+ 383us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 22:54:38.312976 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 22:54:38.312976 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 22:54:38.313148 (+ 172us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 22:54:38.313223 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0929 22:54:38.313481 (+ 258us) server_negotiation.cc:299] Negotiation successful
0929 22:54:38.313588 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260929 22:54:38.313804 5215 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:54:38.304887 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.147.254:35079 (local address 127.0.0.1:38294)
0929 22:54:38.305069 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0929 22:54:38.305106 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0929 22:54:38.305600 (+ 494us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 22:54:38.305844 (+ 244us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 22:54:38.305847 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 22:54:38.306126 (+ 279us) client_negotiation.cc:189] Negotiated authn=SASL
0929 22:54:38.306369 (+ 243us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 22:54:38.306375 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 22:54:38.307053 (+ 678us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 22:54:38.307054 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 22:54:38.307921 (+ 867us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 22:54:38.307925 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 22:54:38.308289 (+ 364us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 22:54:38.308290 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 22:54:38.308409 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 22:54:38.308902 (+ 493us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 22:54:38.308911 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0929 22:54:38.310225 (+ 1314us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 22:54:38.312119 (+ 1894us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 22:54:38.312132 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 22:54:38.312135 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 22:54:38.312436 (+ 301us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 22:54:38.312712 (+ 276us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 22:54:38.312715 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 22:54:38.312715 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 22:54:38.312891 (+ 176us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 22:54:38.313214 (+ 323us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 22:54:38.313218 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 22:54:38.313359 (+ 141us) client_negotiation.cc:769] Sending connection context
0929 22:54:38.313448 (+ 89us) client_negotiation.cc:240] Negotiation successful
0929 22:54:38.313602 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":143,"thread_start_us":94,"threads_started":1}
I20260929 22:54:38.314464 5213 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.14.147.253" port: 37273 } has no permanent_uuid. Determining permanent_uuid...
Sep 29 22:54:38 dist-test-slave-98mk krb5kdc[5056](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790722478, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.147.253@KRBTEST.COM for kudu/127.14.147.253@KRBTEST.COM
WARNING: no policy specified for kudu/127.14.147.252@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.147.252@KRBTEST.COM" created.
I20260929 22:54:38.322558 5215 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:54:38.314790 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.147.253:37273 (local address 127.0.0.1:54936)
0929 22:54:38.314839 (+ 49us) negotiation.cc:107] Waiting for socket to connect
0929 22:54:38.314851 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0929 22:54:38.314895 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 22:54:38.315228 (+ 333us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 22:54:38.315229 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 22:54:38.315402 (+ 173us) client_negotiation.cc:189] Negotiated authn=SASL
0929 22:54:38.315505 (+ 103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 22:54:38.315507 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 22:54:38.316460 (+ 953us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 22:54:38.316462 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 22:54:38.317280 (+ 818us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 22:54:38.317283 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 22:54:38.317855 (+ 572us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 22:54:38.317856 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 22:54:38.317945 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 22:54:38.318370 (+ 425us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 22:54:38.318375 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0929 22:54:38.319788 (+ 1413us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 22:54:38.321580 (+ 1792us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 22:54:38.321581 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 22:54:38.321581 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 22:54:38.321743 (+ 162us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 22:54:38.321918 (+ 175us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 22:54:38.321919 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 22:54:38.321919 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 22:54:38.321967 (+ 48us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 22:54:38.322224 (+ 257us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 22:54:38.322225 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 22:54:38.322262 (+ 37us) client_negotiation.cc:769] Sending connection context
0929 22:54:38.322325 (+ 63us) client_negotiation.cc:240] Negotiation successful
0929 22:54:38.322443 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":31}
I20260929 22:54:38.322587 5217 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:54:38.314865 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54936 (local address 127.14.147.253:37273)
0929 22:54:38.315040 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0929 22:54:38.315044 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 22:54:38.315059 (+ 15us) server_negotiation.cc:407] Connection header received
0929 22:54:38.315076 (+ 17us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 22:54:38.315078 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 22:54:38.315090 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 22:54:38.315200 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0929 22:54:38.315586 (+ 386us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:54:38.316365 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:54:38.317422 (+ 1057us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:54:38.317770 (+ 348us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:54:38.317836 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 22:54:38.319873 (+ 2037us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 22:54:38.319886 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 22:54:38.319886 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 22:54:38.319894 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0929 22:54:38.321480 (+ 1586us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 22:54:38.321814 (+ 334us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 22:54:38.321818 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 22:54:38.321818 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 22:54:38.321866 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 22:54:38.322015 (+ 149us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 22:54:38.322015 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 22:54:38.322016 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 22:54:38.322165 (+ 149us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 22:54:38.322240 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0929 22:54:38.322365 (+ 125us) server_negotiation.cc:299] Negotiation successful
0929 22:54:38.322495 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":64,"server-negotiator.queue_time_us":141,"thread_start_us":103,"threads_started":1}
I20260929 22:54:38.323236 5213 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.14.147.252" port: 45813 } has no permanent_uuid. Determining permanent_uuid...
I20260929 22:54:38.323627 5215 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:54:38.323504 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.147.252:45813 (local address 127.0.0.1:44344)
0929 22:54:38.323547 (+ 43us) negotiation.cc:107] Waiting for socket to connect
0929 22:54:38.323600 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.147.252:45813: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":25}
W20260929 22:54:38.323796 5163 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.14.147.252:45813: connect: Connection refused (error 111)
I20260929 22:54:38.324221 5215 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:54:38.324135 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.147.252:45813 (local address 127.0.0.1:44354)
0929 22:54:38.324174 (+ 39us) negotiation.cc:107] Waiting for socket to connect
0929 22:54:38.324199 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.147.252:45813: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":26}
W20260929 22:54:38.324358 5213 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.14.147.252:45813: Network error: Client connection negotiation failed: client connection to 127.14.147.252:45813: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.147.252 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790722213368738-14927-0/krb5kdc/kudu_127.14.147.252.keytab.
Entry for principal kudu/127.14.147.252 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790722213368738-14927-0/krb5kdc/kudu_127.14.147.252.keytab.
I20260929 22:54:38.336388 14927 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhWOCLk/build/release/bin/kudu
/tmp/dist-test-taskhWOCLk/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790722213368738-14927-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790722213368738-14927-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790722213368738-14927-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790722213368738-14927-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.147.252:45813
--webserver_interface=127.14.147.252
--webserver_port=0
--keytab_file=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790722213368738-14927-0/krb5kdc/kudu_127.14.147.252.keytab
--principal=kudu/127.14.147.252
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.147.212:43383
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.147.254:35079,127.14.147.253:37273,127.14.147.252:45813
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790722213368738-14927-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790722213368738-14927-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790722213368738-14927-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 22:54:38.360317 5137 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.14.147.253" port: 37273 } attempt: 3
Sep 29 22:54:38 dist-test-slave-98mk krb5kdc[5056](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790722478, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.147.254@KRBTEST.COM for kudu/127.14.147.253@KRBTEST.COM
I20260929 22:54:38.366963 5217 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:54:38.360780 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54944 (local address 127.14.147.253:37273)
0929 22:54:38.360850 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0929 22:54:38.360852 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 22:54:38.360861 (+ 9us) server_negotiation.cc:407] Connection header received
0929 22:54:38.360886 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 22:54:38.360888 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 22:54:38.360900 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 22:54:38.360953 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0929 22:54:38.361259 (+ 306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:54:38.361744 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:54:38.362391 (+ 647us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:54:38.362598 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:54:38.362652 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 22:54:38.364724 (+ 2072us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 22:54:38.364726 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 22:54:38.364726 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 22:54:38.364733 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0929 22:54:38.365970 (+ 1237us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 22:54:38.366274 (+ 304us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 22:54:38.366275 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 22:54:38.366275 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 22:54:38.366322 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 22:54:38.366478 (+ 156us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 22:54:38.366479 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 22:54:38.366479 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 22:54:38.366593 (+ 114us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 22:54:38.366648 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0929 22:54:38.366785 (+ 137us) server_negotiation.cc:299] Negotiation successful
0929 22:54:38.366859 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260929 22:54:38.366963 5139 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:54:38.360685 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.147.253:37273 (local address 127.0.0.1:54944)
0929 22:54:38.360738 (+ 53us) negotiation.cc:107] Waiting for socket to connect
0929 22:54:38.360751 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 22:54:38.360792 (+ 41us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 22:54:38.360959 (+ 167us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 22:54:38.360959 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 22:54:38.361118 (+ 159us) client_negotiation.cc:189] Negotiated authn=SASL
0929 22:54:38.361189 (+ 71us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 22:54:38.361191 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 22:54:38.361814 (+ 623us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 22:54:38.361815 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 22:54:38.362320 (+ 505us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 22:54:38.362322 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 22:54:38.362655 (+ 333us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 22:54:38.362656 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 22:54:38.362721 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 22:54:38.363203 (+ 482us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 22:54:38.363207 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0929 22:54:38.364639 (+ 1432us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 22:54:38.366064 (+ 1425us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 22:54:38.366065 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 22:54:38.366066 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 22:54:38.366216 (+ 150us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 22:54:38.366374 (+ 158us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 22:54:38.366374 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 22:54:38.366375 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 22:54:38.366430 (+ 55us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 22:54:38.366655 (+ 225us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 22:54:38.366656 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 22:54:38.366703 (+ 47us) client_negotiation.cc:769] Sending connection context
0929 22:54:38.366758 (+ 55us) client_negotiation.cc:240] Negotiation successful
0929 22:54:38.366857 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":28}
I20260929 22:54:38.367513 5137 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.14.147.252" port: 45813 } has no permanent_uuid. Determining permanent_uuid...
I20260929 22:54:38.367825 5139 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:54:38.367731 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.147.252:45813 (local address 127.0.0.1:44368)
0929 22:54:38.367771 (+ 40us) negotiation.cc:107] Waiting for socket to connect
0929 22:54:38.367799 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.147.252:45813: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":24}
I20260929 22:54:38.368086 5139 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:54:38.368008 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.147.252:45813 (local address 127.0.0.1:44378)
0929 22:54:38.368046 (+ 38us) negotiation.cc:107] Waiting for socket to connect
0929 22:54:38.368067 (+ 21us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.147.252:45813: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":21}
W20260929 22:54:38.368160 5137 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.14.147.252:45813: Network error: Client connection negotiation failed: client connection to 127.14.147.252:45813: connect: Connection refused (error 111)
I20260929 22:54:38.373582 5213 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.14.147.252" port: 45813 } attempt: 1
I20260929 22:54:38.374099 5215 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:54:38.373951 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.147.252:45813 (local address 127.0.0.1:44382)
0929 22:54:38.374015 (+ 64us) negotiation.cc:107] Waiting for socket to connect
0929 22:54:38.374063 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.147.252:45813: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
I20260929 22:54:38.374416 5215 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:54:38.374323 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.147.252:45813 (local address 127.0.0.1:44384)
0929 22:54:38.374363 (+ 40us) negotiation.cc:107] Waiting for socket to connect
0929 22:54:38.374391 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.147.252:45813: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
W20260929 22:54:38.374502 5213 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.14.147.252:45813: Network error: Client connection negotiation failed: client connection to 127.14.147.252:45813: connect: Connection refused (error 111)
I20260929 22:54:38.399345 5137 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.14.147.252" port: 45813 } attempt: 1
I20260929 22:54:38.399887 5139 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:54:38.399759 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.147.252:45813 (local address 127.0.0.1:44386)
0929 22:54:38.399817 (+ 58us) negotiation.cc:107] Waiting for socket to connect
0929 22:54:38.399859 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.147.252:45813: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
I20260929 22:54:38.400236 5139 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:54:38.400154 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.147.252:45813 (local address 127.0.0.1:44402)
0929 22:54:38.400196 (+ 42us) negotiation.cc:107] Waiting for socket to connect
0929 22:54:38.400219 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.147.252:45813: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":28}
W20260929 22:54:38.400331 5137 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.14.147.252:45813: Network error: Client connection negotiation failed: client connection to 127.14.147.252:45813: connect: Connection refused (error 111)
W20260929 22:54:38.419131 5220 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260929 22:54:38.419449 5220 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 22:54:38.419477 5220 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 22:54:38.419517 5220 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 22:54:38.420747 5220 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 22:54:38.420791 5220 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 22:54:38.420809 5220 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 22:54:38.420825 5220 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 22:54:38.420840 5220 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 22:54:38.422370 5220 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.147.212:43383
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790722213368738-14927-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790722213368738-14927-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.14.147.254:35079,127.14.147.253:37273,127.14.147.252:45813
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790722213368738-14927-0/krb5kdc/kudu_127.14.147.252.keytab
--principal=kudu/127.14.147.252
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.147.252:45813
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790722213368738-14927-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.14.147.252
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5220
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790722213368738-14927-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 21:58:54 UTC on ccycloud.quasar-xguoti.root.comops.site
build id 4681390
I20260929 22:54:38.422623 5220 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 22:54:38.422834 5220 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 22:54:38.429523 5228 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 22:54:38.429667 5225 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 22:54:38.429756 5220 server_base.cc:1034] running on GCE node
W20260929 22:54:38.429535 5226 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 22:54:38.430158 5220 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 22:54:38.430461 5220 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 22:54:38.431654 5220 hybrid_clock.cc:648] HybridClock initialized: now 1790722478431628 us; error 41 us; skew 500 ppm
Sep 29 22:54:38 dist-test-slave-98mk krb5kdc[5056](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790722478, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.147.252@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 22:54:38.434118 5220 init.cc:377] Logged in from keytab as kudu/127.14.147.252@KRBTEST.COM (short username kudu)
I20260929 22:54:38.434562 5220 webserver.cc:460] Webserver started at http://127.14.147.252:36159/ using document root <none> and password file <none>
I20260929 22:54:38.434763 5220 fs_manager.cc:359] Metadata directory not provided
I20260929 22:54:38.434809 5220 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 22:54:38.434947 5220 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 22:54:38.436040 5220 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790722213368738-14927-0/minicluster-data/master-2/data/instance:
uuid: "944321c8c8d94939b2b491996a051e46"
format_stamp: "Formatted at 2026-09-29 22:54:38 on dist-test-slave-98mk"
server_key: "054009ae725f146ff8bdbc2a6c3844b8"
server_key_iv: "0a0170bddd62d1bd98c2012a357a43af"
server_key_version: "encryptionkey@0"
I20260929 22:54:38.436372 5220 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790722213368738-14927-0/minicluster-data/master-2/wal/instance:
uuid: "944321c8c8d94939b2b491996a051e46"
format_stamp: "Formatted at 2026-09-29 22:54:38 on dist-test-slave-98mk"
server_key: "054009ae725f146ff8bdbc2a6c3844b8"
server_key_iv: "0a0170bddd62d1bd98c2012a357a43af"
server_key_version: "encryptionkey@0"
I20260929 22:54:38.437633 5220 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260929 22:54:38.438530 5235 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 22:54:38.438782 5220 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260929 22:54:38.438925 5220 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790722213368738-14927-0/minicluster-data/master-2/data,/tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790722213368738-14927-0/minicluster-data/master-2/wal
uuid: "944321c8c8d94939b2b491996a051e46"
format_stamp: "Formatted at 2026-09-29 22:54:38 on dist-test-slave-98mk"
server_key: "054009ae725f146ff8bdbc2a6c3844b8"
server_key_iv: "0a0170bddd62d1bd98c2012a357a43af"
server_key_version: "encryptionkey@0"
I20260929 22:54:38.439028 5220 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790722213368738-14927-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790722213368738-14927-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790722213368738-14927-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 22:54:38.442642 5213 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.14.147.252" port: 45813 } attempt: 2
I20260929 22:54:38.443074 5215 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:54:38.442940 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.147.252:45813 (local address 127.0.0.1:44412)
0929 22:54:38.442998 (+ 58us) negotiation.cc:107] Waiting for socket to connect
0929 22:54:38.443046 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.147.252:45813: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
I20260929 22:54:38.443400 5215 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:54:38.443309 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.147.252:45813 (local address 127.0.0.1:44426)
0929 22:54:38.443348 (+ 39us) negotiation.cc:107] Waiting for socket to connect
0929 22:54:38.443376 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.147.252:45813: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":25}
W20260929 22:54:38.443501 5213 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.14.147.252:45813: Network error: Client connection negotiation failed: client connection to 127.14.147.252:45813: connect: Connection refused (error 111)
I20260929 22:54:38.459553 5220 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 22:54:38.461649 5220 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 22:54:38.461791 5220 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 22:54:38.465679 5220 rpc_server.cc:307] RPC server started. Bound to: 127.14.147.252:45813
I20260929 22:54:38.465724 5287 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.147.252:45813 every 8 connection(s)
I20260929 22:54:38.466064 5220 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790722213368738-14927-0/minicluster-data/master-2/data/info.pb
I20260929 22:54:38.466918 5288 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 22:54:38.468493 5288 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.14.147.254" port: 35079 } has no permanent_uuid. Determining permanent_uuid...
I20260929 22:54:38.471486 14927 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhWOCLk/build/release/bin/kudu as pid 5220
I20260929 22:54:38.471611 14927 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhWOCLk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790722213368738-14927-0/minicluster-data/master-2/wal/instance
I20260929 22:54:38.471792 14927 external_mini_cluster.cc:1428] Setting key 2f6a238458753e45d297960046126e92
Sep 29 22:54:38 dist-test-slave-98mk krb5kdc[5056](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790722478, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.147.252@KRBTEST.COM for kudu/127.14.147.254@KRBTEST.COM
I20260929 22:54:38.475461 5137 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.14.147.252" port: 45813 } attempt: 2
Sep 29 22:54:38 dist-test-slave-98mk krb5kdc[5056](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790722477, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.147.252@KRBTEST.COM
I20260929 22:54:38.478289 5140 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:54:38.469356 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38300 (local address 127.14.147.254:35079)
0929 22:54:38.469485 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0929 22:54:38.469487 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 22:54:38.469787 (+ 300us) server_negotiation.cc:407] Connection header received
0929 22:54:38.470246 (+ 459us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 22:54:38.470247 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 22:54:38.470259 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 22:54:38.470322 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0929 22:54:38.470973 (+ 651us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:54:38.471520 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:54:38.472889 (+ 1369us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:54:38.473187 (+ 298us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:54:38.473243 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 22:54:38.475401 (+ 2158us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 22:54:38.475402 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 22:54:38.475403 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 22:54:38.475410 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0929 22:54:38.476834 (+ 1424us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 22:54:38.477252 (+ 418us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 22:54:38.477253 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 22:54:38.477254 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 22:54:38.477309 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 22:54:38.477564 (+ 255us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 22:54:38.477565 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 22:54:38.477565 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 22:54:38.477707 (+ 142us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 22:54:38.477775 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0929 22:54:38.478117 (+ 342us) server_negotiation.cc:299] Negotiation successful
0929 22:54:38.478198 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260929 22:54:38.478883 5290 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:54:38.469359 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.147.254:35079 (local address 127.0.0.1:38300)
0929 22:54:38.469612 (+ 253us) negotiation.cc:107] Waiting for socket to connect
0929 22:54:38.469655 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0929 22:54:38.470156 (+ 501us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 22:54:38.470384 (+ 228us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 22:54:38.470388 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 22:54:38.470667 (+ 279us) client_negotiation.cc:189] Negotiated authn=SASL
0929 22:54:38.470873 (+ 206us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 22:54:38.470877 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 22:54:38.471629 (+ 752us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 22:54:38.471630 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 22:54:38.472815 (+ 1185us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 22:54:38.472817 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 22:54:38.473259 (+ 442us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 22:54:38.473259 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 22:54:38.473393 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 22:54:38.473909 (+ 516us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 22:54:38.473919 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0929 22:54:38.475319 (+ 1400us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 22:54:38.476916 (+ 1597us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 22:54:38.476925 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 22:54:38.476927 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 22:54:38.477183 (+ 256us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 22:54:38.477362 (+ 179us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 22:54:38.477363 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 22:54:38.477363 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 22:54:38.477493 (+ 130us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 22:54:38.477777 (+ 284us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 22:54:38.477784 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 22:54:38.477945 (+ 161us) client_negotiation.cc:769] Sending connection context
0929 22:54:38.478423 (+ 478us) client_negotiation.cc:240] Negotiation successful
0929 22:54:38.478654 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":194,"mutex_wait_us":24,"thread_start_us":137,"threads_started":1}
I20260929 22:54:38.479488 5288 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.14.147.253" port: 37273 } has no permanent_uuid. Determining permanent_uuid...
Sep 29 22:54:38 dist-test-slave-98mk krb5kdc[5056](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790722478, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.147.254@KRBTEST.COM for kudu/127.14.147.252@KRBTEST.COM
I20260929 22:54:38.481676 5292 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:54:38.472203 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44440 (local address 127.14.147.252:45813)
0929 22:54:38.472374 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0929 22:54:38.472379 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 22:54:38.472403 (+ 24us) server_negotiation.cc:407] Connection header received
0929 22:54:38.472439 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 22:54:38.472441 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 22:54:38.472460 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 22:54:38.472532 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0929 22:54:38.473145 (+ 613us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:54:38.474069 (+ 924us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:54:38.475004 (+ 935us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:54:38.475350 (+ 346us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:54:38.475434 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 22:54:38.478032 (+ 2598us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 22:54:38.478040 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 22:54:38.478040 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 22:54:38.478047 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0929 22:54:38.480292 (+ 2245us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 22:54:38.480611 (+ 319us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 22:54:38.480614 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 22:54:38.480615 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 22:54:38.480689 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 22:54:38.480873 (+ 184us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 22:54:38.480874 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 22:54:38.480874 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 22:54:38.481051 (+ 177us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 22:54:38.481199 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0929 22:54:38.481270 (+ 71us) server_negotiation.cc:299] Negotiation successful
0929 22:54:38.481347 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":136,"thread_start_us":105,"threads_started":1}
I20260929 22:54:38.483238 5139 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:54:38.475710 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.147.252:45813 (local address 127.0.0.1:44450)
0929 22:54:38.475759 (+ 49us) negotiation.cc:107] Waiting for socket to connect
0929 22:54:38.475769 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0929 22:54:38.475806 (+ 37us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 22:54:38.476348 (+ 542us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 22:54:38.476350 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 22:54:38.476520 (+ 170us) client_negotiation.cc:189] Negotiated authn=SASL
0929 22:54:38.476566 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 22:54:38.476568 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 22:54:38.477432 (+ 864us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 22:54:38.477433 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 22:54:38.478152 (+ 719us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 22:54:38.478154 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 22:54:38.478596 (+ 442us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 22:54:38.478597 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 22:54:38.478661 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 22:54:38.479035 (+ 374us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 22:54:38.479040 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0929 22:54:38.480494 (+ 1454us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 22:54:38.482254 (+ 1760us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 22:54:38.482255 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 22:54:38.482255 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 22:54:38.482462 (+ 207us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 22:54:38.482662 (+ 200us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 22:54:38.482663 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 22:54:38.482663 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 22:54:38.482723 (+ 60us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 22:54:38.482945 (+ 222us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 22:54:38.482946 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 22:54:38.482996 (+ 50us) client_negotiation.cc:769] Sending connection context
0929 22:54:38.483054 (+ 58us) client_negotiation.cc:240] Negotiation successful
0929 22:54:38.483149 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":29}
I20260929 22:54:38.483278 5293 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:54:38.475834 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44450 (local address 127.14.147.252:45813)
0929 22:54:38.476039 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0929 22:54:38.476040 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0929 22:54:38.476048 (+ 8us) server_negotiation.cc:407] Connection header received
0929 22:54:38.476067 (+ 19us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 22:54:38.476067 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 22:54:38.476076 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 22:54:38.476147 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0929 22:54:38.476645 (+ 498us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:54:38.477356 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:54:38.478225 (+ 869us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:54:38.478522 (+ 297us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:54:38.478583 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 22:54:38.480597 (+ 2014us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 22:54:38.480598 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 22:54:38.480599 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 22:54:38.480605 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0929 22:54:38.482164 (+ 1559us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 22:54:38.482554 (+ 390us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 22:54:38.482555 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 22:54:38.482555 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 22:54:38.482611 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 22:54:38.482778 (+ 167us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 22:54:38.482778 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 22:54:38.482779 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 22:54:38.482902 (+ 123us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 22:54:38.482957 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0929 22:54:38.483083 (+ 126us) server_negotiation.cc:299] Negotiation successful
0929 22:54:38.483149 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":182,"thread_start_us":107,"threads_started":1}
Sep 29 22:54:38 dist-test-slave-98mk krb5kdc[5056](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790722478, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.147.252@KRBTEST.COM for kudu/127.14.147.253@KRBTEST.COM
I20260929 22:54:38.485337 5137 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d61c2709f5924100bdfd4070c8cc7d75: Bootstrap starting.
I20260929 22:54:38.486107 5137 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d61c2709f5924100bdfd4070c8cc7d75: Neither blocks nor log segments found. Creating new log.
I20260929 22:54:38.486433 5137 log.cc:824] T 00000000000000000000000000000000 P d61c2709f5924100bdfd4070c8cc7d75: Log is configured to *not* fsync() on all Append() calls
I20260929 22:54:38.486806 5217 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:54:38.479809 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54954 (local address 127.14.147.253:37273)
0929 22:54:38.479850 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0929 22:54:38.479851 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0929 22:54:38.479894 (+ 43us) server_negotiation.cc:407] Connection header received
0929 22:54:38.479953 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 22:54:38.479954 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 22:54:38.479964 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 22:54:38.480008 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0929 22:54:38.480365 (+ 357us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:54:38.481010 (+ 645us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:54:38.481896 (+ 886us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:54:38.482147 (+ 251us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:54:38.482210 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 22:54:38.484193 (+ 1983us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 22:54:38.484194 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 22:54:38.484195 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 22:54:38.484202 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0929 22:54:38.485846 (+ 1644us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 22:54:38.486147 (+ 301us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 22:54:38.486148 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 22:54:38.486149 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 22:54:38.486203 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 22:54:38.486358 (+ 155us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 22:54:38.486359 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 22:54:38.486359 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 22:54:38.486464 (+ 105us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 22:54:38.486524 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0929 22:54:38.486654 (+ 130us) server_negotiation.cc:299] Negotiation successful
0929 22:54:38.486721 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260929 22:54:38.486805 5290 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:54:38.479746 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.147.253:37273 (local address 127.0.0.1:54954)
0929 22:54:38.479851 (+ 105us) negotiation.cc:107] Waiting for socket to connect
0929 22:54:38.479862 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0929 22:54:38.479907 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 22:54:38.480009 (+ 102us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 22:54:38.480010 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 22:54:38.480238 (+ 228us) client_negotiation.cc:189] Negotiated authn=SASL
0929 22:54:38.480277 (+ 39us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 22:54:38.480279 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 22:54:38.481099 (+ 820us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 22:54:38.481100 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 22:54:38.481811 (+ 711us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 22:54:38.481814 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 22:54:38.482214 (+ 400us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 22:54:38.482215 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 22:54:38.482276 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 22:54:38.482707 (+ 431us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 22:54:38.482711 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0929 22:54:38.484089 (+ 1378us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 22:54:38.485916 (+ 1827us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 22:54:38.485917 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 22:54:38.485918 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 22:54:38.486086 (+ 168us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 22:54:38.486251 (+ 165us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 22:54:38.486252 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 22:54:38.486252 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 22:54:38.486307 (+ 55us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 22:54:38.486516 (+ 209us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 22:54:38.486517 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 22:54:38.486559 (+ 42us) client_negotiation.cc:769] Sending connection context
0929 22:54:38.486620 (+ 61us) client_negotiation.cc:240] Negotiation successful
0929 22:54:38.486711 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":90}
I20260929 22:54:38.487176 5288 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.14.147.252" port: 45813 } has no permanent_uuid. Determining permanent_uuid...
I20260929 22:54:38.487283 5137 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d61c2709f5924100bdfd4070c8cc7d75: No bootstrap required, opened a new log
Sep 29 22:54:38 dist-test-slave-98mk krb5kdc[5056](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790722477, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.147.254@KRBTEST.COM
I20260929 22:54:38.489162 5137 raft_consensus.cc:348] T 00000000000000000000000000000000 P d61c2709f5924100bdfd4070c8cc7d75 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d61c2709f5924100bdfd4070c8cc7d75" member_type: VOTER last_known_addr { host: "127.14.147.254" port: 35079 } } peers { permanent_uuid: "30054c4a503947e1b79aaa615e7ca7a1" member_type: VOTER last_known_addr { host: "127.14.147.253" port: 37273 } } peers { permanent_uuid: "944321c8c8d94939b2b491996a051e46" member_type: VOTER last_known_addr { host: "127.14.147.252" port: 45813 } }
I20260929 22:54:38.489288 5137 raft_consensus.cc:729] T 00000000000000000000000000000000 P d61c2709f5924100bdfd4070c8cc7d75 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d61c2709f5924100bdfd4070c8cc7d75, State: Initialized, Role: FOLLOWER
I20260929 22:54:38.489372 5137 consensus_queue.cc:260] T 00000000000000000000000000000000 P d61c2709f5924100bdfd4070c8cc7d75 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d61c2709f5924100bdfd4070c8cc7d75" member_type: VOTER last_known_addr { host: "127.14.147.254" port: 35079 } } peers { permanent_uuid: "30054c4a503947e1b79aaa615e7ca7a1" member_type: VOTER last_known_addr { host: "127.14.147.253" port: 37273 } } peers { permanent_uuid: "944321c8c8d94939b2b491996a051e46" member_type: VOTER last_known_addr { host: "127.14.147.252" port: 45813 } }
I20260929 22:54:38.489640 5295 sys_catalog.cc:455] T 00000000000000000000000000000000 P d61c2709f5924100bdfd4070c8cc7d75 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d61c2709f5924100bdfd4070c8cc7d75" member_type: VOTER last_known_addr { host: "127.14.147.254" port: 35079 } } peers { permanent_uuid: "30054c4a503947e1b79aaa615e7ca7a1" member_type: VOTER last_known_addr { host: "127.14.147.253" port: 37273 } } peers { permanent_uuid: "944321c8c8d94939b2b491996a051e46" member_type: VOTER last_known_addr { host: "127.14.147.252" port: 45813 } } }
I20260929 22:54:38.489718 5137 sys_catalog.cc:564] T 00000000000000000000000000000000 P d61c2709f5924100bdfd4070c8cc7d75 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 22:54:38.489801 5295 sys_catalog.cc:458] T 00000000000000000000000000000000 P d61c2709f5924100bdfd4070c8cc7d75 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 22:54:38.490460 5140 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:54:38.482654 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38314 (local address 127.14.147.254:35079)
0929 22:54:38.482795 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0929 22:54:38.482797 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 22:54:38.482803 (+ 6us) server_negotiation.cc:407] Connection header received
0929 22:54:38.482818 (+ 15us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 22:54:38.482819 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 22:54:38.482828 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 22:54:38.482874 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0929 22:54:38.483452 (+ 578us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:54:38.484131 (+ 679us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:54:38.485016 (+ 885us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:54:38.485214 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:54:38.485264 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 22:54:38.487872 (+ 2608us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 22:54:38.487874 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 22:54:38.487874 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 22:54:38.487881 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0929 22:54:38.489352 (+ 1471us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 22:54:38.489726 (+ 374us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 22:54:38.489727 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 22:54:38.489728 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 22:54:38.489781 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 22:54:38.489962 (+ 181us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 22:54:38.489962 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 22:54:38.489963 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 22:54:38.490101 (+ 138us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 22:54:38.490241 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0929 22:54:38.490309 (+ 68us) server_negotiation.cc:299] Negotiation successful
0929 22:54:38.490381 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":123}
Sep 29 22:54:38 dist-test-slave-98mk krb5kdc[5056](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790722478, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.147.252@KRBTEST.COM for kudu/127.14.147.252@KRBTEST.COM
W20260929 22:54:38.492316 5306 catalog_manager.cc:1558] T 00000000000000000000000000000000 P d61c2709f5924100bdfd4070c8cc7d75: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 22:54:38.492374 5306 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260929 22:54:38.494503 5290 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:54:38.487456 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.147.252:45813 (local address 127.0.0.1:44464)
0929 22:54:38.487547 (+ 91us) negotiation.cc:107] Waiting for socket to connect
0929 22:54:38.487556 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0929 22:54:38.487591 (+ 35us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 22:54:38.487729 (+ 138us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 22:54:38.487729 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 22:54:38.487865 (+ 136us) client_negotiation.cc:189] Negotiated authn=SASL
0929 22:54:38.487897 (+ 32us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 22:54:38.487898 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 22:54:38.488722 (+ 824us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 22:54:38.488723 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 22:54:38.489442 (+ 719us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 22:54:38.489444 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 22:54:38.489919 (+ 475us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 22:54:38.489919 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 22:54:38.489965 (+ 46us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 22:54:38.490511 (+ 546us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 22:54:38.490516 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0929 22:54:38.491806 (+ 1290us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 22:54:38.493466 (+ 1660us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 22:54:38.493467 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 22:54:38.493467 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 22:54:38.493649 (+ 182us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 22:54:38.493814 (+ 165us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 22:54:38.493815 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 22:54:38.493815 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 22:54:38.493859 (+ 44us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 22:54:38.494155 (+ 296us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 22:54:38.494156 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 22:54:38.494188 (+ 32us) client_negotiation.cc:769] Sending connection context
0929 22:54:38.494242 (+ 54us) client_negotiation.cc:240] Negotiation successful
0929 22:54:38.494397 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":78}
I20260929 22:54:38.494789 5293 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:54:38.487520 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44464 (local address 127.14.147.252:45813)
0929 22:54:38.487642 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0929 22:54:38.487644 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 22:54:38.487649 (+ 5us) server_negotiation.cc:407] Connection header received
0929 22:54:38.487661 (+ 12us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 22:54:38.487662 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 22:54:38.487669 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 22:54:38.487700 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0929 22:54:38.487969 (+ 269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:54:38.488644 (+ 675us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:54:38.489523 (+ 879us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:54:38.489827 (+ 304us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:54:38.489878 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 22:54:38.491887 (+ 2009us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 22:54:38.491888 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 22:54:38.491888 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 22:54:38.491894 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0929 22:54:38.493396 (+ 1502us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 22:54:38.493717 (+ 321us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 22:54:38.493718 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 22:54:38.493718 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 22:54:38.493764 (+ 46us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 22:54:38.493905 (+ 141us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 22:54:38.493906 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 22:54:38.493906 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 22:54:38.494084 (+ 178us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 22:54:38.494136 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0929 22:54:38.494637 (+ 501us) server_negotiation.cc:299] Negotiation successful
0929 22:54:38.494717 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":111}
I20260929 22:54:38.492887 5069 master_runner.cc:186] Error getting master registration for 127.14.147.253:37273: OK, instance_id { permanent_uuid: "30054c4a503947e1b79aaa615e7ca7a1" instance_seqno: 1790722478299305 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260929 22:54:38.495200 5069 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260929 22:54:38.496982 5288 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 944321c8c8d94939b2b491996a051e46: Bootstrap starting.
Sep 29 22:54:38 dist-test-slave-98mk krb5kdc[5056](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790722477, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.147.253@KRBTEST.COM
I20260929 22:54:38.497730 5288 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 944321c8c8d94939b2b491996a051e46: Neither blocks nor log segments found. Creating new log.
I20260929 22:54:38.497997 5288 log.cc:824] T 00000000000000000000000000000000 P 944321c8c8d94939b2b491996a051e46: Log is configured to *not* fsync() on all Append() calls
I20260929 22:54:38.498654 5288 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 944321c8c8d94939b2b491996a051e46: No bootstrap required, opened a new log
I20260929 22:54:38.500403 5288 raft_consensus.cc:348] T 00000000000000000000000000000000 P 944321c8c8d94939b2b491996a051e46 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d61c2709f5924100bdfd4070c8cc7d75" member_type: VOTER last_known_addr { host: "127.14.147.254" port: 35079 } } peers { permanent_uuid: "30054c4a503947e1b79aaa615e7ca7a1" member_type: VOTER last_known_addr { host: "127.14.147.253" port: 37273 } } peers { permanent_uuid: "944321c8c8d94939b2b491996a051e46" member_type: VOTER last_known_addr { host: "127.14.147.252" port: 45813 } }
I20260929 22:54:38.500532 5288 raft_consensus.cc:729] T 00000000000000000000000000000000 P 944321c8c8d94939b2b491996a051e46 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 944321c8c8d94939b2b491996a051e46, State: Initialized, Role: FOLLOWER
I20260929 22:54:38.500612 5288 consensus_queue.cc:260] T 00000000000000000000000000000000 P 944321c8c8d94939b2b491996a051e46 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d61c2709f5924100bdfd4070c8cc7d75" member_type: VOTER last_known_addr { host: "127.14.147.254" port: 35079 } } peers { permanent_uuid: "30054c4a503947e1b79aaa615e7ca7a1" member_type: VOTER last_known_addr { host: "127.14.147.253" port: 37273 } } peers { permanent_uuid: "944321c8c8d94939b2b491996a051e46" member_type: VOTER last_known_addr { host: "127.14.147.252" port: 45813 } }
I20260929 22:54:38.500955 5217 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:54:38.491487 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54962 (local address 127.14.147.253:37273)
0929 22:54:38.491534 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0929 22:54:38.491536 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 22:54:38.491690 (+ 154us) server_negotiation.cc:407] Connection header received
0929 22:54:38.491713 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 22:54:38.491714 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 22:54:38.491723 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 22:54:38.491761 (+ 38us) server_negotiation.cc:226] Negotiated authn=SASL
0929 22:54:38.492655 (+ 894us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:54:38.493352 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:54:38.494257 (+ 905us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:54:38.495001 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:54:38.495065 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 22:54:38.497753 (+ 2688us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 22:54:38.497754 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 22:54:38.497755 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 22:54:38.497763 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0929 22:54:38.499447 (+ 1684us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 22:54:38.499821 (+ 374us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 22:54:38.499823 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 22:54:38.499823 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 22:54:38.499885 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 22:54:38.500086 (+ 201us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 22:54:38.500087 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 22:54:38.500087 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 22:54:38.500231 (+ 144us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 22:54:38.500386 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0929 22:54:38.500782 (+ 396us) server_negotiation.cc:299] Negotiation successful
0929 22:54:38.500860 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260929 22:54:38.500973 5309 sys_catalog.cc:455] T 00000000000000000000000000000000 P 944321c8c8d94939b2b491996a051e46 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d61c2709f5924100bdfd4070c8cc7d75" member_type: VOTER last_known_addr { host: "127.14.147.254" port: 35079 } } peers { permanent_uuid: "30054c4a503947e1b79aaa615e7ca7a1" member_type: VOTER last_known_addr { host: "127.14.147.253" port: 37273 } } peers { permanent_uuid: "944321c8c8d94939b2b491996a051e46" member_type: VOTER last_known_addr { host: "127.14.147.252" port: 45813 } } }
I20260929 22:54:38.501355 5309 sys_catalog.cc:458] T 00000000000000000000000000000000 P 944321c8c8d94939b2b491996a051e46 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 22:54:38.501488 5288 sys_catalog.cc:564] T 00000000000000000000000000000000 P 944321c8c8d94939b2b491996a051e46 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 22:54:38.502462 14927 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 22:54:38.503368 5111 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "d61c2709f5924100bdfd4070c8cc7d75"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38314
I20260929 22:54:38.503553 5111 raft_consensus.cc:482] T 00000000000000000000000000000000 P d61c2709f5924100bdfd4070c8cc7d75 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260929 22:54:38.503608 5111 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d61c2709f5924100bdfd4070c8cc7d75 [term 0 FOLLOWER]: Advancing to term 1
W20260929 22:54:38.503711 5320 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 944321c8c8d94939b2b491996a051e46: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 22:54:38.503801 5320 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260929 22:54:38.504433 5111 raft_consensus.cc:504] T 00000000000000000000000000000000 P d61c2709f5924100bdfd4070c8cc7d75 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d61c2709f5924100bdfd4070c8cc7d75" member_type: VOTER last_known_addr { host: "127.14.147.254" port: 35079 } } peers { permanent_uuid: "30054c4a503947e1b79aaa615e7ca7a1" member_type: VOTER last_known_addr { host: "127.14.147.253" port: 37273 } } peers { permanent_uuid: "944321c8c8d94939b2b491996a051e46" member_type: VOTER last_known_addr { host: "127.14.147.252" port: 45813 } }
I20260929 22:54:38.504686 5111 leader_election.cc:288] T 00000000000000000000000000000000 P d61c2709f5924100bdfd4070c8cc7d75 [CANDIDATE]: Term 1 election: Requested vote from peers 30054c4a503947e1b79aaa615e7ca7a1 (127.14.147.253:37273), 944321c8c8d94939b2b491996a051e46 (127.14.147.252:45813)
I20260929 22:54:38.505409 5262 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "d61c2709f5924100bdfd4070c8cc7d75" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "944321c8c8d94939b2b491996a051e46"
I20260929 22:54:38.505515 5262 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 944321c8c8d94939b2b491996a051e46 [term 0 FOLLOWER]: Advancing to term 1
I20260929 22:54:38.506392 5262 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 944321c8c8d94939b2b491996a051e46 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d61c2709f5924100bdfd4070c8cc7d75 in term 1.
I20260929 22:54:38.506597 5087 leader_election.cc:302] T 00000000000000000000000000000000 P d61c2709f5924100bdfd4070c8cc7d75 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 944321c8c8d94939b2b491996a051e46, d61c2709f5924100bdfd4070c8cc7d75; no voters:
I20260929 22:54:38.506738 5295 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d61c2709f5924100bdfd4070c8cc7d75 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 22:54:38.506886 5295 raft_consensus.cc:686] T 00000000000000000000000000000000 P d61c2709f5924100bdfd4070c8cc7d75 [term 1 LEADER]: Becoming Leader. State: Replica: d61c2709f5924100bdfd4070c8cc7d75, State: Running, Role: LEADER
I20260929 22:54:38.506983 5295 consensus_queue.cc:237] T 00000000000000000000000000000000 P d61c2709f5924100bdfd4070c8cc7d75 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d61c2709f5924100bdfd4070c8cc7d75" member_type: VOTER last_known_addr { host: "127.14.147.254" port: 35079 } } peers { permanent_uuid: "30054c4a503947e1b79aaa615e7ca7a1" member_type: VOTER last_known_addr { host: "127.14.147.253" port: 37273 } } peers { permanent_uuid: "944321c8c8d94939b2b491996a051e46" member_type: VOTER last_known_addr { host: "127.14.147.252" port: 45813 } }
I20260929 22:54:38.507403 5295 sys_catalog.cc:455] T 00000000000000000000000000000000 P d61c2709f5924100bdfd4070c8cc7d75 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d61c2709f5924100bdfd4070c8cc7d75. Latest consensus state: current_term: 1 leader_uuid: "d61c2709f5924100bdfd4070c8cc7d75" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d61c2709f5924100bdfd4070c8cc7d75" member_type: VOTER last_known_addr { host: "127.14.147.254" port: 35079 } } peers { permanent_uuid: "30054c4a503947e1b79aaa615e7ca7a1" member_type: VOTER last_known_addr { host: "127.14.147.253" port: 37273 } } peers { permanent_uuid: "944321c8c8d94939b2b491996a051e46" member_type: VOTER last_known_addr { host: "127.14.147.252" port: 45813 } } }
I20260929 22:54:38.507503 5295 sys_catalog.cc:458] T 00000000000000000000000000000000 P d61c2709f5924100bdfd4070c8cc7d75 [sys.catalog]: This master's current role is: LEADER
I20260929 22:54:38.506965 5220 master_runner.cc:186] Error getting master registration for 127.14.147.253:37273: OK, instance_id { permanent_uuid: "30054c4a503947e1b79aaa615e7ca7a1" instance_seqno: 1790722478299305 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260929 22:54:38.508832 5220 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260929 22:54:38.517783 5331 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 22:54:38.518059 5331 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 22:54:38.524999 5293 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:54:38.507244 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44470 (local address 127.14.147.252:45813)
0929 22:54:38.507279 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0929 22:54:38.507280 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0929 22:54:38.515588 (+ 8308us) server_negotiation.cc:407] Connection header received
0929 22:54:38.515684 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 22:54:38.515685 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 22:54:38.515698 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 22:54:38.515749 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0929 22:54:38.517237 (+ 1488us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:54:38.518024 (+ 787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:54:38.519099 (+ 1075us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:54:38.519405 (+ 306us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:54:38.519483 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 22:54:38.522081 (+ 2598us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 22:54:38.522083 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 22:54:38.522084 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 22:54:38.522091 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0929 22:54:38.523638 (+ 1547us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 22:54:38.524136 (+ 498us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 22:54:38.524137 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 22:54:38.524138 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 22:54:38.524194 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 22:54:38.524375 (+ 181us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 22:54:38.524375 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 22:54:38.524376 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 22:54:38.524508 (+ 132us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 22:54:38.524595 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0929 22:54:38.524798 (+ 203us) server_negotiation.cc:299] Negotiation successful
0929 22:54:38.524883 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260929 22:54:38.526449 5140 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:54:38.507042 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38328 (local address 127.14.147.254:35079)
0929 22:54:38.507082 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0929 22:54:38.507083 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0929 22:54:38.515755 (+ 8672us) server_negotiation.cc:407] Connection header received
0929 22:54:38.515810 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 22:54:38.515811 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 22:54:38.515823 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 22:54:38.515888 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0929 22:54:38.516840 (+ 952us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:54:38.517580 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:54:38.519225 (+ 1645us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:54:38.519535 (+ 310us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:54:38.519580 (+ 45us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 22:54:38.522636 (+ 3056us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 22:54:38.522637 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 22:54:38.522638 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 22:54:38.522646 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0929 22:54:38.524607 (+ 1961us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 22:54:38.525360 (+ 753us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 22:54:38.525361 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 22:54:38.525361 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 22:54:38.525435 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 22:54:38.525727 (+ 292us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 22:54:38.525728 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 22:54:38.525728 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 22:54:38.525870 (+ 142us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 22:54:38.525938 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0929 22:54:38.526262 (+ 324us) server_negotiation.cc:299] Negotiation successful
0929 22:54:38.526356 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260929 22:54:38.526943 5217 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:54:38.505735 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54966 (local address 127.14.147.253:37273)
0929 22:54:38.505786 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0929 22:54:38.505788 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 22:54:38.511670 (+ 5882us) server_negotiation.cc:407] Connection header received
0929 22:54:38.511781 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 22:54:38.511782 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 22:54:38.511795 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 22:54:38.511864 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0929 22:54:38.512619 (+ 755us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:54:38.513343 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:54:38.514410 (+ 1067us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:54:38.514741 (+ 331us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:54:38.514811 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 22:54:38.519864 (+ 5053us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 22:54:38.519866 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 22:54:38.519867 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 22:54:38.519875 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0929 22:54:38.521396 (+ 1521us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 22:54:38.523722 (+ 2326us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 22:54:38.523723 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 22:54:38.523724 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 22:54:38.523791 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 22:54:38.526166 (+ 2375us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 22:54:38.526168 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 22:54:38.526168 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 22:54:38.526303 (+ 135us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 22:54:38.526369 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0929 22:54:38.526764 (+ 395us) server_negotiation.cc:299] Negotiation successful
0929 22:54:38.526851 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260929 22:54:38.528312 5262 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 944321c8c8d94939b2b491996a051e46 [term 1 FOLLOWER]: Refusing update from remote peer d61c2709f5924100bdfd4070c8cc7d75: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 22:54:38.528621 5327 consensus_queue.cc:1035] T 00000000000000000000000000000000 P d61c2709f5924100bdfd4070c8cc7d75 [LEADER]: Connected to new peer: Peer: permanent_uuid: "944321c8c8d94939b2b491996a051e46" member_type: VOTER last_known_addr { host: "127.14.147.252" port: 45813 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 22:54:38.529973 5309 sys_catalog.cc:455] T 00000000000000000000000000000000 P 944321c8c8d94939b2b491996a051e46 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d61c2709f5924100bdfd4070c8cc7d75. Latest consensus state: current_term: 1 leader_uuid: "d61c2709f5924100bdfd4070c8cc7d75" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d61c2709f5924100bdfd4070c8cc7d75" member_type: VOTER last_known_addr { host: "127.14.147.254" port: 35079 } } peers { permanent_uuid: "30054c4a503947e1b79aaa615e7ca7a1" member_type: VOTER last_known_addr { host: "127.14.147.253" port: 37273 } } peers { permanent_uuid: "944321c8c8d94939b2b491996a051e46" member_type: VOTER last_known_addr { host: "127.14.147.252" port: 45813 } } }
I20260929 22:54:38.530076 5309 sys_catalog.cc:458] T 00000000000000000000000000000000 P 944321c8c8d94939b2b491996a051e46 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 22:54:38.531158 5295 sys_catalog.cc:455] T 00000000000000000000000000000000 P d61c2709f5924100bdfd4070c8cc7d75 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "d61c2709f5924100bdfd4070c8cc7d75" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d61c2709f5924100bdfd4070c8cc7d75" member_type: VOTER last_known_addr { host: "127.14.147.254" port: 35079 } } peers { permanent_uuid: "30054c4a503947e1b79aaa615e7ca7a1" member_type: VOTER last_known_addr { host: "127.14.147.253" port: 37273 } } peers { permanent_uuid: "944321c8c8d94939b2b491996a051e46" member_type: VOTER last_known_addr { host: "127.14.147.252" port: 45813 } } }
I20260929 22:54:38.531246 5295 sys_catalog.cc:458] T 00000000000000000000000000000000 P d61c2709f5924100bdfd4070c8cc7d75 [sys.catalog]: This master's current role is: LEADER
I20260929 22:54:38.531373 5309 sys_catalog.cc:455] T 00000000000000000000000000000000 P 944321c8c8d94939b2b491996a051e46 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "d61c2709f5924100bdfd4070c8cc7d75" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d61c2709f5924100bdfd4070c8cc7d75" member_type: VOTER last_known_addr { host: "127.14.147.254" port: 35079 } } peers { permanent_uuid: "30054c4a503947e1b79aaa615e7ca7a1" member_type: VOTER last_known_addr { host: "127.14.147.253" port: 37273 } } peers { permanent_uuid: "944321c8c8d94939b2b491996a051e46" member_type: VOTER last_known_addr { host: "127.14.147.252" port: 45813 } } }
I20260929 22:54:38.531497 5309 sys_catalog.cc:458] T 00000000000000000000000000000000 P 944321c8c8d94939b2b491996a051e46 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 22:54:38.531719 5333 mvcc.cc:205] Tried to move back new op lower bound from 7334799272049602560 to 7334799271965384704. Current Snapshot: MvccSnapshot[applied={T|T < 7334799272049602560}]
I20260929 22:54:38.534634 5213 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.14.147.252" port: 45813 } attempt: 3
Sep 29 22:54:38 dist-test-slave-98mk krb5kdc[5056](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790722478, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.147.253@KRBTEST.COM for kudu/127.14.147.252@KRBTEST.COM
I20260929 22:54:38.543628 5215 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:54:38.534898 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.147.252:45813 (local address 127.0.0.1:44472)
0929 22:54:38.534946 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0929 22:54:38.534957 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0929 22:54:38.535001 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 22:54:38.535243 (+ 242us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 22:54:38.535244 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 22:54:38.535446 (+ 202us) client_negotiation.cc:189] Negotiated authn=SASL
0929 22:54:38.535501 (+ 55us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 22:54:38.535503 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 22:54:38.536424 (+ 921us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 22:54:38.536425 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 22:54:38.537251 (+ 826us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 22:54:38.537253 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 22:54:38.537728 (+ 475us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 22:54:38.537729 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 22:54:38.537791 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 22:54:38.538296 (+ 505us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 22:54:38.538301 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0929 22:54:38.540431 (+ 2130us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 22:54:38.542509 (+ 2078us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 22:54:38.542510 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 22:54:38.542511 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 22:54:38.542731 (+ 220us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 22:54:38.542938 (+ 207us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 22:54:38.542938 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 22:54:38.542938 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 22:54:38.543007 (+ 69us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 22:54:38.543276 (+ 269us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 22:54:38.543277 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 22:54:38.543325 (+ 48us) client_negotiation.cc:769] Sending connection context
0929 22:54:38.543396 (+ 71us) client_negotiation.cc:240] Negotiation successful
0929 22:54:38.543531 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":30}
I20260929 22:54:38.543638 5293 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:54:38.535069 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44472 (local address 127.14.147.252:45813)
0929 22:54:38.535113 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0929 22:54:38.535129 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0929 22:54:38.535136 (+ 7us) server_negotiation.cc:407] Connection header received
0929 22:54:38.535158 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 22:54:38.535159 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 22:54:38.535169 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 22:54:38.535227 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0929 22:54:38.535579 (+ 352us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:54:38.536340 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:54:38.537335 (+ 995us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:54:38.537655 (+ 320us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:54:38.537723 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 22:54:38.540517 (+ 2794us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 22:54:38.540518 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 22:54:38.540518 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 22:54:38.540526 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0929 22:54:38.542409 (+ 1883us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 22:54:38.542803 (+ 394us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 22:54:38.542804 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 22:54:38.542804 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 22:54:38.542870 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 22:54:38.543078 (+ 208us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 22:54:38.543078 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 22:54:38.543079 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 22:54:38.543219 (+ 140us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 22:54:38.543276 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0929 22:54:38.543462 (+ 186us) server_negotiation.cc:299] Negotiation successful
0929 22:54:38.543544 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260929 22:54:38.543931 5331 catalog_manager.cc:1347] Generated new cluster ID: 8aeb21bcefe4400199f7044cfdde2f9d
I20260929 22:54:38.543977 5331 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 22:54:38.545610 5213 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 30054c4a503947e1b79aaa615e7ca7a1: Bootstrap starting.
I20260929 22:54:38.546335 5213 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 30054c4a503947e1b79aaa615e7ca7a1: Neither blocks nor log segments found. Creating new log.
I20260929 22:54:38.546607 5213 log.cc:824] T 00000000000000000000000000000000 P 30054c4a503947e1b79aaa615e7ca7a1: Log is configured to *not* fsync() on all Append() calls
I20260929 22:54:38.548048 5213 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 30054c4a503947e1b79aaa615e7ca7a1: No bootstrap required, opened a new log
I20260929 22:54:38.550071 5213 raft_consensus.cc:348] T 00000000000000000000000000000000 P 30054c4a503947e1b79aaa615e7ca7a1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d61c2709f5924100bdfd4070c8cc7d75" member_type: VOTER last_known_addr { host: "127.14.147.254" port: 35079 } } peers { permanent_uuid: "30054c4a503947e1b79aaa615e7ca7a1" member_type: VOTER last_known_addr { host: "127.14.147.253" port: 37273 } } peers { permanent_uuid: "944321c8c8d94939b2b491996a051e46" member_type: VOTER last_known_addr { host: "127.14.147.252" port: 45813 } }
I20260929 22:54:38.550213 5213 raft_consensus.cc:729] T 00000000000000000000000000000000 P 30054c4a503947e1b79aaa615e7ca7a1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 30054c4a503947e1b79aaa615e7ca7a1, State: Initialized, Role: FOLLOWER
I20260929 22:54:38.550288 5213 consensus_queue.cc:260] T 00000000000000000000000000000000 P 30054c4a503947e1b79aaa615e7ca7a1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d61c2709f5924100bdfd4070c8cc7d75" member_type: VOTER last_known_addr { host: "127.14.147.254" port: 35079 } } peers { permanent_uuid: "30054c4a503947e1b79aaa615e7ca7a1" member_type: VOTER last_known_addr { host: "127.14.147.253" port: 37273 } } peers { permanent_uuid: "944321c8c8d94939b2b491996a051e46" member_type: VOTER last_known_addr { host: "127.14.147.252" port: 45813 } }
I20260929 22:54:38.550617 5213 sys_catalog.cc:564] T 00000000000000000000000000000000 P 30054c4a503947e1b79aaa615e7ca7a1 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 22:54:38.551860 5186 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 30054c4a503947e1b79aaa615e7ca7a1 [term 0 FOLLOWER]: Advancing to term 1
I20260929 22:54:38.552771 5186 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 30054c4a503947e1b79aaa615e7ca7a1 [term 1 FOLLOWER]: Refusing update from remote peer d61c2709f5924100bdfd4070c8cc7d75: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 22:54:38.553081 5327 consensus_queue.cc:1035] T 00000000000000000000000000000000 P d61c2709f5924100bdfd4070c8cc7d75 [LEADER]: Connected to new peer: Peer: permanent_uuid: "30054c4a503947e1b79aaa615e7ca7a1" member_type: VOTER last_known_addr { host: "127.14.147.253" port: 37273 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 22:54:38.553154 5187 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "d61c2709f5924100bdfd4070c8cc7d75" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "30054c4a503947e1b79aaa615e7ca7a1"
I20260929 22:54:38.553900 5187 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 30054c4a503947e1b79aaa615e7ca7a1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d61c2709f5924100bdfd4070c8cc7d75 in term 1.
I20260929 22:54:38.557332 5350 sys_catalog.cc:455] T 00000000000000000000000000000000 P 30054c4a503947e1b79aaa615e7ca7a1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d61c2709f5924100bdfd4070c8cc7d75" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d61c2709f5924100bdfd4070c8cc7d75" member_type: VOTER last_known_addr { host: "127.14.147.254" port: 35079 } } peers { permanent_uuid: "30054c4a503947e1b79aaa615e7ca7a1" member_type: VOTER last_known_addr { host: "127.14.147.253" port: 37273 } } peers { permanent_uuid: "944321c8c8d94939b2b491996a051e46" member_type: VOTER last_known_addr { host: "127.14.147.252" port: 45813 } } }
I20260929 22:54:38.557456 5350 sys_catalog.cc:458] T 00000000000000000000000000000000 P 30054c4a503947e1b79aaa615e7ca7a1 [sys.catalog]: This master's current role is: FOLLOWER
W20260929 22:54:38.557914 5349 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 30054c4a503947e1b79aaa615e7ca7a1: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
I20260929 22:54:38.557888 5295 sys_catalog.cc:455] T 00000000000000000000000000000000 P d61c2709f5924100bdfd4070c8cc7d75 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "d61c2709f5924100bdfd4070c8cc7d75" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d61c2709f5924100bdfd4070c8cc7d75" member_type: VOTER last_known_addr { host: "127.14.147.254" port: 35079 } } peers { permanent_uuid: "30054c4a503947e1b79aaa615e7ca7a1" member_type: VOTER last_known_addr { host: "127.14.147.253" port: 37273 } } peers { permanent_uuid: "944321c8c8d94939b2b491996a051e46" member_type: VOTER last_known_addr { host: "127.14.147.252" port: 45813 } } }
W20260929 22:54:38.557955 5349 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260929 22:54:38.557969 5295 sys_catalog.cc:458] T 00000000000000000000000000000000 P d61c2709f5924100bdfd4070c8cc7d75 [sys.catalog]: This master's current role is: LEADER
I20260929 22:54:38.557372 5353 sys_catalog.cc:455] T 00000000000000000000000000000000 P 30054c4a503947e1b79aaa615e7ca7a1 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "d61c2709f5924100bdfd4070c8cc7d75" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d61c2709f5924100bdfd4070c8cc7d75" member_type: VOTER last_known_addr { host: "127.14.147.254" port: 35079 } } peers { permanent_uuid: "30054c4a503947e1b79aaa615e7ca7a1" member_type: VOTER last_known_addr { host: "127.14.147.253" port: 37273 } } peers { permanent_uuid: "944321c8c8d94939b2b491996a051e46" member_type: VOTER last_known_addr { host: "127.14.147.252" port: 45813 } } }
I20260929 22:54:38.558665 5353 sys_catalog.cc:458] T 00000000000000000000000000000000 P 30054c4a503947e1b79aaa615e7ca7a1 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 22:54:38.559356 5338 sys_catalog.cc:455] T 00000000000000000000000000000000 P 30054c4a503947e1b79aaa615e7ca7a1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d61c2709f5924100bdfd4070c8cc7d75. Latest consensus state: current_term: 1 leader_uuid: "d61c2709f5924100bdfd4070c8cc7d75" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d61c2709f5924100bdfd4070c8cc7d75" member_type: VOTER last_known_addr { host: "127.14.147.254" port: 35079 } } peers { permanent_uuid: "30054c4a503947e1b79aaa615e7ca7a1" member_type: VOTER last_known_addr { host: "127.14.147.253" port: 37273 } } peers { permanent_uuid: "944321c8c8d94939b2b491996a051e46" member_type: VOTER last_known_addr { host: "127.14.147.252" port: 45813 } } }
I20260929 22:54:38.559458 5338 sys_catalog.cc:458] T 00000000000000000000000000000000 P 30054c4a503947e1b79aaa615e7ca7a1 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 22:54:38.559574 5331 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 22:54:38.561215 5331 catalog_manager.cc:1504] Loading token signing keys...
I20260929 22:54:38.563740 5351 mvcc.cc:205] Tried to move back new op lower bound from 7334799272150171648 to 7334799271965384704. Current Snapshot: MvccSnapshot[applied={T|T < 7334799272150171648 or (T in {7334799272150171648})}]
I20260929 22:54:38.571375 5331 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d61c2709f5924100bdfd4070c8cc7d75: Generated new TSK 0
I20260929 22:54:38.571651 5331 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260929 22:54:38.595952 5140 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 22:54:38.593986 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38342 (local address 127.14.147.254:35079)
0929 22:54:38.594037 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0929 22:54:38.594039 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 22:54:38.594986 (+ 947us) server_negotiation.cc:407] Connection header received
0929 22:54:38.595202 (+ 216us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 22:54:38.595204 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 22:54:38.595215 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 22:54:38.595245 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0929 22:54:38.595879 (+ 634us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38342: BlockingRecv error: recv got EOF from 127.0.0.1:38342 (error 108)
Metrics: {"server-negotiator.queue_time_us":26}
W20260929 22:54:38.596586 5293 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 22:54:38.594223 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44480 (local address 127.14.147.252:45813)
0929 22:54:38.594262 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0929 22:54:38.594263 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0929 22:54:38.595092 (+ 829us) server_negotiation.cc:407] Connection header received
0929 22:54:38.596119 (+ 1027us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 22:54:38.596120 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 22:54:38.596133 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 22:54:38.596163 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0929 22:54:38.596527 (+ 364us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:44480: BlockingRecv error: recv got EOF from 127.0.0.1:44480 (error 108)
Metrics: {"server-negotiator.queue_time_us":19}
W20260929 22:54:38.597254 5217 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 22:54:38.594438 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54972 (local address 127.14.147.253:37273)
0929 22:54:38.594499 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0929 22:54:38.594500 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0929 22:54:38.595364 (+ 864us) server_negotiation.cc:407] Connection header received
0929 22:54:38.596797 (+ 1433us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 22:54:38.596798 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 22:54:38.596809 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 22:54:38.596837 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0929 22:54:38.597182 (+ 345us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54972: BlockingRecv error: recv got EOF from 127.0.0.1:54972 (error 108)
Metrics: {"server-negotiator.queue_time_us":20}
W20260929 22:54:38.599754 5217 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 22:54:38.599002 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54980 (local address 127.14.147.253:37273)
0929 22:54:38.599037 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0929 22:54:38.599039 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 22:54:38.599226 (+ 187us) server_negotiation.cc:407] Connection header received
0929 22:54:38.599324 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 22:54:38.599325 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 22:54:38.599343 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 22:54:38.599370 (+ 27us) server_negotiation.cc:226] Negotiated authn=SASL
0929 22:54:38.599708 (+ 338us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54980: BlockingRecv error: recv got EOF from 127.0.0.1:54980 (error 108)
Metrics: {"server-negotiator.queue_time_us":18}
W20260929 22:54:38.604373 5217 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 22:54:38.601998 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54988 (local address 127.14.147.253:37273)
0929 22:54:38.602033 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0929 22:54:38.602035 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 22:54:38.603831 (+ 1796us) server_negotiation.cc:407] Connection header received
0929 22:54:38.603930 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 22:54:38.603932 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 22:54:38.603942 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 22:54:38.603971 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0929 22:54:38.604318 (+ 347us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54988: BlockingRecv error: recv got EOF from 127.0.0.1:54988 (error 108)
Metrics: {"server-negotiator.queue_time_us":18}
W20260929 22:54:38.608529 5217 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 22:54:38.607710 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55000 (local address 127.14.147.253:37273)
0929 22:54:38.607744 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0929 22:54:38.607746 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 22:54:38.607997 (+ 251us) server_negotiation.cc:407] Connection header received
0929 22:54:38.608094 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 22:54:38.608095 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 22:54:38.608105 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 22:54:38.608134 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0929 22:54:38.608465 (+ 331us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:55000: BlockingRecv error: recv got EOF from 127.0.0.1:55000 (error 108)
Metrics: {"server-negotiator.queue_time_us":16}
W20260929 22:54:38.614528 5217 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 22:54:38.613661 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55006 (local address 127.14.147.253:37273)
0929 22:54:38.613701 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0929 22:54:38.613703 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 22:54:38.613962 (+ 259us) server_negotiation.cc:407] Connection header received
0929 22:54:38.614062 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 22:54:38.614063 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 22:54:38.614074 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 22:54:38.614101 (+ 27us) server_negotiation.cc:226] Negotiated authn=SASL
0929 22:54:38.614460 (+ 359us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:55006: BlockingRecv error: recv got EOF from 127.0.0.1:55006 (error 108)
Metrics: {"server-negotiator.queue_time_us":19}
W20260929 22:54:38.624913 5217 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 22:54:38.623950 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55014 (local address 127.14.147.253:37273)
0929 22:54:38.624009 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0929 22:54:38.624011 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 22:54:38.624293 (+ 282us) server_negotiation.cc:407] Connection header received
0929 22:54:38.624393 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 22:54:38.624394 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 22:54:38.624406 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 22:54:38.624436 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0929 22:54:38.624835 (+ 399us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:55014: BlockingRecv error: recv got EOF from 127.0.0.1:55014 (error 108)
Metrics: {"server-negotiator.queue_time_us":23}
W20260929 22:54:38.643356 5217 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 22:54:38.642369 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55018 (local address 127.14.147.253:37273)
0929 22:54:38.642415 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0929 22:54:38.642417 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 22:54:38.642755 (+ 338us) server_negotiation.cc:407] Connection header received
0929 22:54:38.642860 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 22:54:38.642861 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 22:54:38.642873 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 22:54:38.642902 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0929 22:54:38.643295 (+ 393us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:55018: BlockingRecv error: recv got EOF from 127.0.0.1:55018 (error 108)
Metrics: {"server-negotiator.queue_time_us":22}
W20260929 22:54:38.677320 5217 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 22:54:38.676643 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55034 (local address 127.14.147.253:37273)
0929 22:54:38.676696 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0929 22:54:38.676698 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 22:54:38.676809 (+ 111us) server_negotiation.cc:407] Connection header received
0929 22:54:38.676866 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 22:54:38.676867 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 22:54:38.676878 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 22:54:38.676928 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0929 22:54:38.677254 (+ 326us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:55034: BlockingRecv error: recv got EOF from 127.0.0.1:55034 (error 108)
Metrics: {"server-negotiator.queue_time_us":29}
W20260929 22:54:38.743219 5217 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 22:54:38.742440 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55040 (local address 127.14.147.253:37273)
0929 22:54:38.742491 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0929 22:54:38.742493 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 22:54:38.742505 (+ 12us) server_negotiation.cc:407] Connection header received
0929 22:54:38.742583 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 22:54:38.742584 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 22:54:38.742594 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 22:54:38.742637 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0929 22:54:38.743090 (+ 453us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:55040: BlockingRecv error: recv got EOF from 127.0.0.1:55040 (error 108)
Metrics: {"server-negotiator.queue_time_us":24}
W20260929 22:54:38.872812 5217 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 22:54:38.872187 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55054 (local address 127.14.147.253:37273)
0929 22:54:38.872252 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0929 22:54:38.872264 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0929 22:54:38.872336 (+ 72us) server_negotiation.cc:407] Connection header received
0929 22:54:38.872389 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 22:54:38.872390 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 22:54:38.872401 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 22:54:38.872450 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0929 22:54:38.872748 (+ 298us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:55054: BlockingRecv error: recv got EOF from 127.0.0.1:55054 (error 108)
Metrics: {"server-negotiator.queue_time_us":27}
W20260929 22:54:39.130831 5217 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 22:54:39.130164 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55070 (local address 127.14.147.253:37273)
0929 22:54:39.130231 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0929 22:54:39.130233 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 22:54:39.130243 (+ 10us) server_negotiation.cc:407] Connection header received
0929 22:54:39.130314 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 22:54:39.130314 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 22:54:39.130325 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 22:54:39.130366 (+ 41us) server_negotiation.cc:226] Negotiated authn=SASL
0929 22:54:39.130779 (+ 413us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:55070: BlockingRecv error: recv got EOF from 127.0.0.1:55070 (error 108)
Metrics: {"server-negotiator.queue_time_us":38}
I20260929 22:54:39.504313 5320 catalog_manager.cc:1259] Loaded cluster ID: 8aeb21bcefe4400199f7044cfdde2f9d
I20260929 22:54:39.504395 5320 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 944321c8c8d94939b2b491996a051e46: loading cluster ID for follower catalog manager: success
I20260929 22:54:39.505215 5320 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 944321c8c8d94939b2b491996a051e46: acquiring CA information for follower catalog manager: success
I20260929 22:54:39.505496 5320 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 944321c8c8d94939b2b491996a051e46: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260929 22:54:39.558439 5349 catalog_manager.cc:1259] Loaded cluster ID: 8aeb21bcefe4400199f7044cfdde2f9d
I20260929 22:54:39.558519 5349 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 30054c4a503947e1b79aaa615e7ca7a1: loading cluster ID for follower catalog manager: success
I20260929 22:54:39.559320 5349 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 30054c4a503947e1b79aaa615e7ca7a1: acquiring CA information for follower catalog manager: success
I20260929 22:54:39.559645 5349 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 30054c4a503947e1b79aaa615e7ca7a1: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260929 22:54:39.646381 5420 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:54:39.643978 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55080 (local address 127.14.147.253:37273)
0929 22:54:39.644087 (+ 109us) server_negotiation.cc:206] Beginning negotiation
0929 22:54:39.644088 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0929 22:54:39.644098 (+ 10us) server_negotiation.cc:407] Connection header received
0929 22:54:39.644140 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 22:54:39.644141 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 22:54:39.644191 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 22:54:39.644254 (+ 63us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 22:54:39.644425 (+ 171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:54:39.644924 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:54:39.645746 (+ 822us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:54:39.645938 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:54:39.645994 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 22:54:39.646102 (+ 108us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 22:54:39.646210 (+ 108us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 22:54:39.646263 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0929 22:54:39.646276 (+ 13us) server_negotiation.cc:299] Negotiation successful
0929 22:54:39.646294 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":90,"thread_start_us":51,"threads_started":1}
I20260929 22:54:39.651029 5426 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:54:39.648032 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44492 (local address 127.14.147.252:45813)
0929 22:54:39.648144 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0929 22:54:39.648145 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0929 22:54:39.648153 (+ 8us) server_negotiation.cc:407] Connection header received
0929 22:54:39.648190 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 22:54:39.648190 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 22:54:39.648229 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 22:54:39.648299 (+ 70us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 22:54:39.648587 (+ 288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:54:39.649066 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:54:39.649969 (+ 903us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:54:39.650200 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:54:39.650288 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 22:54:39.650545 (+ 257us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 22:54:39.650686 (+ 141us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 22:54:39.650777 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0929 22:54:39.650927 (+ 150us) server_negotiation.cc:299] Negotiation successful
0929 22:54:39.650952 (+ 25us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86,"thread_start_us":55,"threads_started":1}
I20260929 22:54:39.655490 5420 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:54:39.653010 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55090 (local address 127.14.147.253:37273)
0929 22:54:39.653053 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0929 22:54:39.653055 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 22:54:39.653147 (+ 92us) server_negotiation.cc:407] Connection header received
0929 22:54:39.653193 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 22:54:39.653194 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 22:54:39.653202 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 22:54:39.653248 (+ 46us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 22:54:39.653545 (+ 297us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:54:39.654050 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:54:39.654801 (+ 751us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:54:39.655025 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:54:39.655088 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 22:54:39.655209 (+ 121us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 22:54:39.655241 (+ 32us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 22:54:39.655287 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0929 22:54:39.655347 (+ 60us) server_negotiation.cc:299] Negotiation successful
0929 22:54:39.655371 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260929 22:54:39.655917 5426 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:54:39.652953 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44504 (local address 127.14.147.252:45813)
0929 22:54:39.652992 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0929 22:54:39.652994 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 22:54:39.653258 (+ 264us) server_negotiation.cc:407] Connection header received
0929 22:54:39.653289 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 22:54:39.653290 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 22:54:39.653300 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 22:54:39.653355 (+ 55us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 22:54:39.653495 (+ 140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:54:39.653988 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:54:39.655076 (+ 1088us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:54:39.655372 (+ 296us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:54:39.655579 (+ 207us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 22:54:39.655609 (+ 30us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 22:54:39.655643 (+ 34us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 22:54:39.655682 (+ 39us) server_negotiation.cc:1035] Waiting for connection context
0929 22:54:39.655820 (+ 138us) server_negotiation.cc:299] Negotiation successful
0929 22:54:39.655843 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260929 22:54:39.656728 5434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 22:54:39.652988 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38344 (local address 127.14.147.254:35079)
0929 22:54:39.653104 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0929 22:54:39.653106 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 22:54:39.653365 (+ 259us) server_negotiation.cc:407] Connection header received
0929 22:54:39.653420 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 22:54:39.653421 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 22:54:39.653476 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 22:54:39.653557 (+ 81us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 22:54:39.654187 (+ 630us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:54:39.654943 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:54:39.655821 (+ 878us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 22:54:39.656236 (+ 415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 22:54:39.656295 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 22:54:39.656453 (+ 158us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 22:54:39.656541 (+ 88us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 22:54:39.656581 (+ 40us) server_negotiation.cc:1035] Waiting for connection context
0929 22:54:39.656627 (+ 46us) server_negotiation.cc:299] Negotiation successful
0929 22:54:39.656650 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":97,"thread_start_us":58,"threads_started":1}
I20260929 22:54:39.658174 14927 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhWOCLk/build/release/bin/kudu with pid 5069
I20260929 22:54:39.662720 14927 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhWOCLk/build/release/bin/kudu with pid 5144
I20260929 22:54:39.667196 14927 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhWOCLk/build/release/bin/kudu with pid 5220
2026-09-29T22:54:39Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1 (2724 ms)
[----------] 2 tests from ConnectToFollowerMasterTest (5452 ms total)
[----------] Global test environment tear-down
[==========] 56 tests from 4 test suites ran. (266307 ms total)
[ PASSED ] 55 tests.
[ FAILED ] 1 test, listed below:
[ FAILED ] SecurityITest.TestJwtMiniCluster
1 FAILED TEST